2010 International Symposium on Nonlinear Theory and its Applications
NOLTA2010, Krakow, Poland, September 5-8, 2010

Facial Feature Animation and Its Artistic Representation
Yang Yang†‡ , Yuanqi Su† , Yuehu Liu† and Yoshifumi Nishio‡†Institute of Artificial Intelligence and Robotics
Xi’an Jiaotong University, Xi’an 710049, China‡Dept. of Electrical and Electronic Engineering
Tokushima University, Tokushima 770-8506, Japan
Email: {yyang, yqsu, liuyh}@aiar.xjtu.edu.cn, nishio@ee.tokushima-u.ac.jp
Abstract—Facial animation and artistic representation
are active research topics in the multimedia industry. This
paper addresses facial object with geometric feature to generate expressions using the artistic representation. Firstly,
we propose an animation method to transfer expressions
from a source face to a target face automatically, which is
eﬀective when only a single face with neutral expression
is available as the novel target. Then, we provide a facial
artistic representation framework and exhibit its producing
results by facial cartoon and sketches. Combining above
work by building the corresponding map between the facial
animation and representation features, we generate expressions for the target face vividly.
1. Introduction
Facial feature animation and its artistic representation
has played an important role in the human’s lives withincreasing requirements in the entertainment industry andmultimedia communication. Recently, the movie Avatar
made a huge success by using the related techniques. It
took us to a spectacular world by exhibiting real actors’
performance onto fanciful characters in the movie. Meanwhile, more and more communication tools in the daily
life, like the MSN and mobile phone, show their potential
applications for the personalized interface. To generate theartistic facial animation will be welcomed by every user. In
this paper, we firstly animate facial features by transferring
expressions from a source performer’s face to target faces,
and then exhibit the animation results using the artistic representation.
The commonly used animation technique in the movie
industry needs animators to repeatedly define facial animation parameters for every face. In addition, it often requires
complex computing and manual adjusting because of thecorrelations among these parameters. Some blendshapemethods [1]-[3] were proposed based on key examples ofthe source and target faces. They simplified the animation
process just to estimate and transfer the blending weights.
However, it is also a tedious work to prepare key examples
for every face. In this paper, our animation method directlytransfers and adjusts motion vectors of the source face totarget faces, which can animate the novel target face eﬀectively.

Some impressive works have been developed to produce facial artistic representations. PicToon [4] utilizeda statistic model with non-parametric sampling to create
cartoon from the given face image. It can generate the facial sketches and specific animations from templates. The
MSN cartoon [5] was an online application published byMicrosoft. Human’s interactions are required by the system to select and adjust the personalized cartoon shape andaccessories. A facial sketch generation method was proposed by Xu et al. [6], which gave a hierarchical facial
representation for rendering facial sketches. In our early
work about a face cartoon producer [7], a cartoon rendering method was proposed based on the automatic templatefeature fitting and adaptation. In this paper, we extend the
producer to generate artistic animation results by the corresponding map built between the facial animation and representation features.
2. System Overview
The system overview is shown in Fig. 1. When input
a target face image with neutral expression, a set of feature points are extracted by the ASM [8] method. We canrender an artistic representation for this target face by theartistic face producer. For generating expression animation
sequences, source expressions are transferred to target facial features firstly, and then the whole animation results
are represented with the same style as the artistic target
face. In the facial expression database, examples are labeled oﬀ-line on various expressions of diﬀerent people. In
the artistic face database, rendering templates are drawn byartists. In the following sections, we describe each function
block in the facial feature animation and representation in
more detail.
3. Expression Transfer for Facial Feature Animation
Human’s faces share a common topological structureand exhibit a similar movement when making the same expression. When just one single face is available as the target
input, Blanz et al. [9] gave an eﬀective expression transfer
method by the vector space operations. Their method isbased on the assumption that the 3D displacements of surface points are the same for all individuals, which is just- 213 -

Artistic Face
Producer

FeatureExtraction

Target face
image

Animation
Sequence
Generation

Facial Feature
Animation

Sourceexpressions

Artistic
target face

Artistic
Representation

Facial Expression
Database

Generation
results

Artistic Face
Database

Figure 1: The system overview.
suited to very similar faces. We propose a two steps expression transfer method. Inspired by Blanz’s work [9],we could firstly acquire a coarse animation result using thedirect motion vector’s transferring. Although that may generate an unreasonable face, it has mapped the source face’s
movement on the target face roughly. And then we adjust
this coarse animation result according to the personality ofthe target face.
3.1. Global Movement Transferring
Facial feature is represented by a set of feature points’coordinates C = (x1 , y1 , x2 , y2 , ..., xN , yN ), where N is thepoints’ number. We suppose that the first frame in source
expression sequence is always with the neutral expression.
Motion vectors are calculated by the source neutral face
CS N and expressional face CS E , and transferred to the target
facial feature CT N directly as:
CT E = CT N + CS E − CS N .

(1)Without the magnitude and direction adjusting, the transfer result CT E is easily to cause an unreasonable face. To
improve it, we project CT E in the facial space and reconstruct it by principal component analysis (PCA) model:′

CT E = C̄ + Pb,(2)

where, C̄ is the mean shape of training examples and P contains unit eigenvectors of the covariance matrix. b is theprincipal component for a given CT E . The reconstructed′
CT E is the most similar reasonable face as CT E for the reason of the PCA.
One problem is that the PCA model is learned based on
various faces in the database, which is hard to maintain the
personality especially from a coarse input. Moreover, themotion vectors of the source face also need to be adapted
for the target face. In the following part, since a globally reasonable facial structure could be guaranteed by the
PCA, we discuss the local feature adjustment for each facial component.

3.2. Local Personalized Adjustment
Xiong et al. [10] proposed a facial component modelusing neighbor reconstruction method for personalized expression synthesis. In the facial parameter space, they usedneighbor examples to reconstruct target face, and transferred reconstruction weights for generating target expressional faces. However, every expression of the source faceshould be known in advance by the animator, since theyneeded re-preparing corresponding expression’s examples.
Here, we use the neighbor reconstruction as a constraint toformulate the personalized adjustment as an optimization
problem.
We construct the facial expression database by a setof example pairs. Every example pair contains one neutral face and one expressional face belonged to the same
person. All facial features are separated into seven local
subsets according to facial components, including left andright eyes, left and right eyebrows, nose, mouth and facecontour.
Firstly, each feature subset of target face finds the neighbor examples by the k-nearest neighbor algorithm fromneutral faces in the example pairs.
And then, we calculate the reconstruct weights for local component based on two principles: one is the reconstructed similarity with target neutral face; the other is tomatch with the current expression. Using the global move′
ment transferring result CT E as the current expressionalface, the optimization function is built as follows:′min(∥CTcE −c
αis.t. αci ≥ 0 ,

K
∑
i=1
K
∑

i=1

c 2
αci S iE
∥ + λ∥CTc N −

K
∑
i=1

c 2
αci S iN∥)(3)

αci = 1.

We use c ∈ (1, 2, ..., 7) as the index for seven facial compoc
cnents to represent facial feature subset. S iN
and S iE
are theneutral and expressional facial feature subset belonged toith example pair in the K neighbors. Parameter λ balances
two terms in Eq. (3).
This is a quadratic problem with linear constraints,
where the objective function is positive semidefinite. We- 214 -

use fmincon function in Matlab to solve Eq. (3) and acquire a set of optimized weights α∗ci .
Finally, the fine animation result CT∗cE is acquired by reconstructing from neighbor expressional examples using
weights α∗c
i :
K
∑
c
CT∗cE =α∗c
(4)i S iE .

the animation sequence is (xi , yi ). To build correspondence
mapping between two set of features, we use thin-plate
splines (TPS) [12] in the following form for 2D points.
fu (x, y) = d1 (x, y) +fv (x, y) = d2 (x, y) +i=1

4. Artistic Facial Representation
Our prior work [7] proposed a face cartoon producer
for automatically rendering cartoon based on extracted features from static input face image. This producer can beeasily extended to a multiple styles artistic face producer
by just changing the rendering templates. Liu et al. [11]presented a three-layer framework for cartoon facial animation, but did not discuss the animation method for thesequence generation. In this section, we briefly introduce
the framework of artistic face producer and then give thedetails about the animation sequence generation.
4.1. Artistic Face Producer
Given an input face image and extracted feature points,
the process for generating artistic presentation has the following steps.
Template selection: A group of rendering templates are
drawn by artists. All these templates are parameterizedwith rendering vector’s information, rendering rules, facialgeometric features, and so on. For the target face, the mostsimilar template is selected by the facial features’ comparison.
Model adjusting: The selected template is similar to thetarget face, but still needs to be adapted for exactly fitting
the target facial feature by the geometric deformation algorithm.
Artistic Rendering: The producer combines the painting entities with the picked out rendering rules and spatial
arrangements of the template, and outputs feature representation with artistic style in a vector way.
With above steps, facial features could be artisticallypresented correspondingly. Two classes of artistic face rendering results, facial cartoon and sketches, are shown in
Fig. 2.
4.2. Animation Sequence Generation

N
∑
i=1
N
∑
i=1

ω1i U(∥(x, y)T − (ui , vi )T ∥)(5)
ω2i U(∥(x, y)T − (ui , vi )T ∥).

Here, U(r) = r2 ln r is the kernel function in the TPS, drepresents the aﬃne transformation, and ωi represents the
non-aﬃne deformation. All the coeﬃcients are solved by aminimized energy function as defined in [12].
The reason for choosing TPS as the mapping function is
its suitability in the facial deformation. There is a smooth∫∫ 2
2∂2 f 2) + ( ∂∂y2f )2 ]d xdy inness measure term β [( ∂∂x2f )2 + 2( ∂x∂y
the minimized energy function for the TPS fitting, which isin charged by the parameter β. We can control the smoothness of the transformation by diﬀerent β flexibly. During
people making expressions, rigid transformation dominates
the eyebrow’s motions, so that we choose a bigger β to animate eyebrow; but for the mouth, a smaller β is used for its
variable shape.
At the end, we repeat the third step in the artistic face
producer to render the entity in each frame for generating
the final animation sequence. Some animation results areartistically presented in Fig. 3, in which the source expressions are collected from the Cohn-Kanade database [13].
5. Conclusion
In this paper, we presented a facial feature animation andartistic representation method, which can generate expression animation results with artistic style. Our method issuited for the novel input face image. No complex model
parameters need to be adjusted, which saves a lot of manual work for the application. We extended our artistic face
producer by the corresponding map built between the facial animation and representation features. The animation
results were presented vividly.
Acknowledgments
This work was supported by the National Natural Science Foundation of China under Grant No. 60775017 andthe National Basic Research Program of China under Grant
No. 2007CB311005.

The rendering style should be consistent to all the frames
in the expression sequence for one target face. Facial features in the animation sequence have been acquired in thesection 3. Here, we build the corresponding map between
the facial animation and representation features. Then therendering vectors of the artistic target face are warped toeach animation result correspondingly.
Suppose the facial feature in artistic target face is represented as (ui , vi ), and facial feature of an arbitrary frame in- 215 -

References[1] I. Buck and F. Adam, “Performance-driven handdrawn animation,” Proc. Int’l Symp. on Nonphotorealistic Aniamtion and Rendering, pp. 101108, 2000.
[2] E. Chuang and C. Bregler, “Performance driven
facial animation using blendshape interpolatio,”Target face
image

Artistic
representation

Figure 2: Facial artistic representation.

Sourceexpressions

Generation
results

Figure 3: Animation results with facial sketches representation.
Standford University Computer Science Technical
Report, CSTR-2002-02, 2002.

[8] T.F. Cootes and C.J. Taylor, “Active shape modelstheir training and application,” Computer Vision andImage Understanding, vol. 61, no. 1, pp. 38-59,
1995.

[3] X. Liu, T. Mao, et al., “Facial animation by optimized blendshapes from motion capture data,” Computer Animation and Virtual Worlds, vol. 19, no. 3-4,
pp. 235-245, 2008.
[4] H. Chen, Z. Liu, C. Rose, Y. Xu, H.Y. Shum,and D. Salesin, “Example-based composite sketching of human portraits,” Proc. Int’l Symp. on Nonphotorealistic Aniamtion and Rendering, pp. 95101, 2004.
[5] MSN cartoon, http://cartoon.msn.com.cn.
[6] Z. Xu, H. Chen, S. Zhu, and J. Luo, “A hierarchical compositional model for face representationand sketching,” IEEE Trans. on Pattern Analysis andMachine Intelligence, vol. 30, no. 6, pp. 955-969,
2008.
[7] Y. Liu, Y. Su, Y. Shao, Z. Wu, and Y. Yang, “A facecartoon producer for digital content service,” Proc.
Int’l Workshop of Mobile Multimedia Processing,pp. 188-202, 2008.

[9] V. Blanz, C. Basso, et al., “Reanimating faces in images and video,” Proc. EUROGRAPHICS, pp. 641650, 2003.
[10] L. Xiong, N. Zheng, et al., “Facial expression synthesis based on facial component model,” Int’l J.
Pattern Recognition and Artificial Intelligence, vol.
23, no. 3, pp. 637-657, 2009.
[11] Y. Liu, Y. Su, et al., “A facial sketch animation
generation for mobile communication,” Proc. IAPR
Conf. on Machine Vision Applications, pp. 532-535,
2007.
[12] F.L. Bookstein, “Principal warps: thin-plate splines
and the decomposition of deformations,” IEEE
Trans. on Pattern Analysis and Machine Intelligence, vol. 11, no. 6, pp. 567-585, 1989.
[13] T. Kanade, J. F. Cohn, and Y. Tian, “Comprehensive
database for facial expression analysis,” Proc. Automatic Face and Gesture Recognition, pp. 46-53,
2000.

- 216 -