A Robust Method of Facial Feature Tracking for Moving Images
Yuka Nomura*Takayuki Itoh †Yasushi Yamaguchi ‡Graduate School of Interdisciplinary Information

Graduate School of Humanitics and

The Interfaculty Initiative in Information

Studies, The University of Tokyo

Sciences, Ochanomizu University

Studies, The University of Tokyo

Abstract

However, those techniques have problems that elaboration

Robustness of feature tracking is really important to recognize

and robustness for human’s motions are not enough. When we

facial expressions and movements of human in the natural state.

consider the human in the natural state, we also have toThis paper presents a technique for a robust facial feature

consider sudden occlusions and various human’s motions.

tracking for the human facial videos. Our technique first

Robustness of feature tracking is really important to recognize

detects particular feature points by template matching, and thennatural facial expressions and movements. Even though

tracks the points in moving images by optical flow estimation.

various robust techniques for feature tracking are proposed

At the same time, the technique determines the reliabilities of[1,2,3], we need to discuss for development of more efficientthe detected feature points. If the technique determines atechniques.

feature point is not reliable, it executes template matching inThis paper presents a technique for a robust facial feature

the neighbor domain or estimates the position of the feature

tracking for the human facial videos, which realizes continuous

point according to the level of the reliability. By those steps,

tracking even while the features are occluded. Our techniquewhen the difficulty to detect the feature point is dissolved, weapplies template matching to the first frame of facial video forcan detect and track feature points again appropriately andinitialization of feature detection. While it determines thequickly. The technique can be applied to understanding offeature points are reliable, it tracks them by applying opticaluser's delicate feelings, conditions, or psychology.

flow estimation since it is a fast feature tracking method. In

Keywords: Facial feature tracking, template matching, optical

addition, the technique supports an exception handling

flow estimation, video processing

mechanism which is applied when the technique failed to track
the points. The exception handling mechanism first executes

1．Introduction

template matching to a partial area. If the result of template

Various kinds of techniques for recognition of facial

matching is proper, it determines that it is a proper feature

expressions have been recently presented. Those techniques

point, and returns to tracking by the optical flow estimation.

realize better human and computer interaction. In most of the

Otherwise, it tracks other reliable points estimated by position

techniques, computers recognize facial changes by tracking

measurement, so that it can quickly return to the step of partial

face areas, facial parts and features. Those techniques willtemplate matching when it comes to discover the point again.

realize that computers understand human’s facial expression,

By applying those steps, our technique realizes robust and fast

psychology, gestures, and so on. Some researches havemethod to detect and track facial feature points.

attempted to recognize facial changes by detection of particular

Our technique is somewhat similar to a recent technique [4]facial feature points and tracking them.

which has already realized robust and continuous tracking evenwhile the features are under occlusions. We think our technique
is different from the technique presented in [4] as follows:*e-mail: yuka＠graco.c.u-tokyo.ac.jp•

Our technique applies optical flow estimation while it canproperly track the features, because it is faster than†e-mail: itot＠computer.org

template matching.

‡e-mail: yama＠graco.c.u-tokyo.ac.jp•

Our technique tracks estimated features while they arereally occluded, and it makes quickly return to templatematching when it can be detected again.

features as template images.

The reminder of the paper is organized as follows: Section 2
introduces related works in facial feature detection andtracking.

Section3

describes

overview

and2.

first frame of the moving images.

detailed

implementation of the presented technique. Sections 4 present

Detect particular features by template matching, in the3.

Track the detected feature points by Lucas-Kanade
methods, in the following frames.

the experiments of the technique. Section 5 concludes andIf the detected points are reliable to track, the technique will

suggests future works.

continue to track them. However, if the detected points are not

2. Related Worksreliable, it executes the following steps.

Researches on faces and facial features detection and tracking

1.

regions of points at the previous frame.

have been hot topics．There are techniques focus on structuresand pixels of features. One of the techniques focuses on

Execute the template matching again in the near-field2.

Estimate the appropriate position, if the similarity

features in line-structures of hair and eyebrow and to focus on

between the template and the area extracted by thefeatures in the appearance of pixels corresponding to eye, nose

template matching is low.

and mouth [5]. Face color and dark color regions are also

These two steps correspond to an exception handling on

useful information for facial feature detection [6]. Head pose,

reliabilities of detected feature points. This is why ourfacetechnique can realize a more robust facial feature tracking.

deformation,

andcombinationofvariousfacialmovements are used as keys for recognizing the positions offacial feature points and analyzing various human’s conditions

Basic Tracking

Initialization[7,8]. Some other researches use infrared camera [9] to detectand track eyes, which are also useful for human’s gazeestimation.
Some of several feature tracking techniques are somewhat

Feature points

Feature tracking

initialization

by LK methodby template matchingcommon to our technique, since they use pattern matching for
Exception Handling

detection of features [1,2,3,4]. Especially, some techniques[2,3] apply combination method of pattern matching and shape
extraction to realize robust detection and tracking. One other

Estimate

Detect the feature points

the point’s position

by limited template matching

technique focuses on an optical-flow based approach for facial
feature tracking that is sensitive to subtle changes in facial

Figure 1．Overview flowchart

expression [11].
Robustness of facial feature tracking is an important problem.

3.2 Template matchingSome techniques have already realized the continuous

The main loop of the presented technique tracks important

detection of face under partial occlusions [4,10]. As discussed

facial features to track in the moving images. First, we prepare

in Section 1, the technique described in [4] is somewhat similar

template images to detect features by template matching.

to our technique.

Supposing that the template image is the size of M×N pixel,

There are various applications of facial feature tracking. For

the technique overlaps it onto a partial M×N pixel area of theexample, understanding of user's delicate feelings andmoving images. Here, let I(i,j) as a value of pixel of i-th frompsychology [10] is an interesting application.

the left and j-th from the top in the original image, and T(i,j) as
a value of pixel of i-th from the left and j-th from the top in the

3. Implementation

template image. This step calculates difference R applying the

3.1 Overview

following equation (1) for the template matching.

The technique presented in this paper tracks feature points inmoving images taken by a digital video camera. The technique

R =N −1 M−1∑ ∑ | I ( i , j ) − T ( i , j ) | ...(1)
j= 0

i= 0

applies template matching for facial feature detection, andThe partial area, where the values minimize R, is considered

optical flow for the feature tracking, while it switches them

to be the similar part, and we take it as the detection result of

according to the reliability of the detected points. The

the facial features in the technique.

algorithm consists of the following steps.
1.

Prepare the images of the subject’s particular facial

Currently, we manually extract template images from eachsubject’s face image, and manually specify the position of thetemplate image, as described in Section 3.2.

feature point on each template image.

3.2 Optical flow estimation
The technique then tracks the feature points in the following
frames. We applied Lucas-Kanade method tracks them as an
Figure 2．The first frame of a subject’s facial video

optical flow estimation method. Lucas-Kanade method is oneof the spatial partial optimization methods assumed that an
optical flow becomes fixed in the local domain of the samex

object in the dynamic scene [13]. Let the coordinates of ayposition on an image as (x, y), and the gray value of the image
in time t as I(x, y). The optical flow (u, v) in the local domain w
Figure 3．The template images(right eye side, left eye side, right lip corner, left lip corner)

is defined as the following equation (2). Figure 5 shows anexample of the results of Lucas-Kanade method using a
subject' s facial video.

Table 1 ． Values of feature points and colors of template∑u =matching results’ rectangles
point values (x, y)Color of rectangles

right eye side

(6, 10)Blue

left eye side

(16, 12)Red

right lip corner(6, 5)

Cyan

left lip corner(15, 8)magentaw∂I∂I⋅ [J ( p ) − I ( p ) ]∑ w ∂ y ⋅ [J ( p ) − I ( p ) ]
∂x
,v =
2
2
⎛ ∂I ⎞⎛ ∂I ⎞∑ w ⎜⎝ ∂ x ⎟⎠∑ w ⎜⎜ ∂ y ⎟⎟⎠⎝...(2)I ( p ) = I ( x , y , t ), J ( p ) = I ( x , y , t + ∂ t )Frame 0

Frame 16

Frame 26

Figure 5．Results of feature tracking by Lucas-Kanade method

3.3 Exception handling
Figure 4. The result of template matching(rectangles)
and feature points (green points)

The presented technique estimates reliability of facial featurepoints while tracking features. It determines that the feature
tracking result is unusual, when one of the following situations
occurs:

We show an example of detection of the facial features by・When the feature point is not detected.

the template matching in the facial video provided in the・When the difference of the circumference pixel value of thedatabase in [12]. Original image size is 320x240pixels, eachoriginal feature point and the point after movement exceeds

template image has 18x18pixels. Figure 2 shows a subject inthe threshold.

the first frame, and Figure 3 shows his template images. As

In these cases, the proposal technique executes templateshown in Table 1, we decided beforehand each feature point inmatching limited to the small square domain enclosing theeach template images and each color of rectangles as the

position of the feature point in the previous frame. Current our

template matching results. As a result, it detects corners of theimplementation defines the small domain about three times

eyes and mouth from the first frame in the facial video of alarger than the template images. Consequently, when thesubject as shown in Figure 4.

minimum value of R in equation (1) exceeds the threshold, it

After executing template matching and detecting the similar

considers that the error of the detection result is large, and thepartial area for each template image, the technique will track

presented technique therefore searches for the similar portion

the point which has similar value as the feature point in thein a larger domain. If the minimum value of R becomes smallerthan the threshold, it considers that this detection result is

One is for more proper detection, and the other is for reduction

reliable, and the technique supposes it as a result of the properof computation time. As an example, we experiment on a stilldetection. Figure 6 shows an example flowchart of the process.

face image (266x315 pixels) using the same template images as

If the technique unusually tracks a feature point byshown in Figure 3. Figure 7 shows the original image and theLucas-Kanade method, it defines the small domain (dotted

results of the template matching for whole and limited region.

frame) centering on the former feature point (blue point), as

It took 0.2180 seconds for template matching for whole region,

shown in Figure 6 (Right). Then, the technique executesand 0.0040 seconds for limited region. From the result, ourtemplate matching in the limited domain, and re-detects thetechnique is a more efficient way to detect proper featuresfeature points.

especially for real-time facial video.
When the process cannot detect the proper points, we
suppose that the feature is difficult to detect because it may be
occluded by the front object, or the face looks away. In thesecases, the technique calculates the estimated position of thefeature point based on movements of other feature points.
The technique calculates errors between pni, and Apoi, wherepni is i-th detected feature points in the n-th frame, A is a

Unusual detection(right eye)matrix for projection transformation, and poi is i-th detectedfeature point in the first frame, by the equation (3).
N −1

Define the domain

Re-detect the feature point

E = ∑ w i | Ap 0 i − p ni | 2...(3)i=0⎛a
A = ⎜⎜ 11
⎝ a21

a12
a22

a13 ⎞
⎟ , p ni = ( xni , y ni ,1)
a23 ⎟⎠

Figure 6. Template matching for the limited region

Frame N+2
Re-tracking

Frame N
Define the estimated point

Frame N+1
Template matching for limited area

Figure 8. Exception handling in the case of occlusion

Here N is the number of feature points, and wi is the weight
Figure 7. (Left) Original image. (Upper right) The result of

proportional to the reliability of detection results, and E is the

template matching for whole region. (Lower right) The

error value. Our technique calculates six values of A which

result of template matching for limited region.

minimizes E, and then calculates the estimated position of thelost feature points. Figure 8 shows one of the examples of the

The template matching for limited area has two advantages.

case that the features are difficult to detect, and how theexception handling works in the situation.
By those processing, when occlusions or some difficulties
for feature detection are dissolved, we can detect feature points
again appropriately and quickly by applying template matchingat the near domains of the estimated facial feature points.
Figure 10. The initialization of another subject’s video

4. Experiments
Figure 9 shows the differences between conventional method
and our method using the subject’s facial video (320x240
pixels). We used template images as shown in Figure 3 (18x18

xy

pixels). And in this case, we defined 60x60 pixels as a squaredomain size for the limited template matching. Figure 9 (Left)shows the result of tracking by Lucas-Kanade method, and(3, 10)(12, 10)(4, 8)(11, 8)Figure 9 (Right) shows the result of our technique. For

Figure 11．The template images and values of feature points

clearness these figures show only the feature points unusually(right eye side, left eye side, right lip corner, left lip corner)

tracked by Lucas-Kanade method. In the left figures, green
points show that features, the right corner of eye in frame 15
and the corners of mouth in frame, are detected incorrectly. On
the other hand, we succeeded to detect the features correctly
because the technique executes template matching again, as
shown in Figure 9 (Right). Here colors of the result rectangles
are defined in Table 1, and colors of the detected points are

Frame170

defined in Table 2.

Table 2．Colors of the detected points
Color
Usual tracking points

Green

Points detected by the partial template matching

Yellow

Points detected by the position estimation

Red

Frame229

Frame235
Frame15

Figure 12. Tracking by Lucas Kanade method(Left)and our method(Right) on another subject

Figures 10 to 12 show experiments on another subject’s
results. We took his facial video (352x240pixels) and Figure 11
shows template images (17x17pixels) and the coordinates ofFrame53

the feature points. And we defined the colors of the result
rectangles same as the first subject, as described in Table 1.

Figure 9. Tracking by Lucas Kanade method(Left)and our method(Right)

Around frame 170 of his facial video, his hand occludes his
face. In this case, Figure 12 shows that Lucas-Kanade method
fails to track the points from frame 170 to 235. On the otherhand, our technique executes the exception handling. It shows

Video Database”,

the result of template matching for limited region (cyan

http://synapse.vit.iit.nrc.ca/db/video/faces/cvglab/

rectangle) and the estimated point (red point). In this case,[5] B. Lucas and T. Kanade, “An Iterative Image Registration

since the technique determined the result of template matching

Technique with an Application to Stereo Vision” Proceedings

is not reliable, it estimated the proper point. In frame 229,of the 7th International Joint Conference on Artificial

since the occlusion was released, the technique discovered theIntelligence (IJCAI '81), April, 1981, pp. 674-679.

proper feature point by the template matching, and defined the[6] K. Hayashi , M. Hashimoto and K. Sumi “Head Direction

new feature point (yellow point). After this handling, we

Estimation with Robust and Accurate Tracker to Facial

succeeded to keep tracking features in the following frames, as

Features”, The Institute of Electronics, Information andshown in frame 235.

Communication Engineers, Vol. J84-D-Ⅱ No.8 pp1762-1771,
2001.8.

5. Conclusion[7] K. Fukui and O. Yamaguchi, “Facial Feature Point

This paper presented a robust technique for facial feature

Extraction

tracking which successfully recovers unusual detections. Our

Extraction”, The Institute of Electronics, Information andtechnique realizes a robust facial feature tracking under natural

Communication Engineers, Vol. J80-D-Ⅱ No.8 pp2170-2177,

conditions by considering reliability of detected points. It

1997.8.

applies fast Lucas-Kanade method to track features during[8] H. Ikeda and K. Moriya, “High-precision Facial Region

usual situations, and applies template matching during

Detection Method and Its Utilization”, Meeting on Image

exception handling. We think that this combination is good for

Recognition and Understanding 2006(MIRU2006), pp.418-423,

the balance of computation time and reliability.

2006.7.

Our final goal on this study is application to the presented[9] J. F. Cohn et al. , ” Feature-Point Tracking by Optical Flowtechnique for understanding user's delicate feelings, orDiscriminates Subtle Differences in Facial Expression”,

psychology. For example, we would like to develop following

Proceedings of the 3rd IEEE International Conference on

systems applying the presented technique:

Automatic Face and Gesture Recognition (FG '98), pp.396-401,

MethodBasedonCombination

of

Shape

・The system serving contents which suit user's condition.

1998. 4.

・The avatar which supports user's communication.

[10] H. Shimoda et al. , “A Prototype of a Real-time

Toward the above goal, we would like to experience withExpression Recognition System from Dynamic Facial Image”,

more facial feature points which are important to understand

Human Interface Society Vol.1, No.2, 1999.

human's facial expressions. Moreover, we would like to[11] K. Sugiyama et al. ,“Drowsiness Level Detection bymeasure the processing time of the unusual detection

Measuring Blinks Utilizing Image Processing”, R&D Review

mentioned in Section 3.2 and analyze the accuracy of aof Toyota CRDL, Vol. 31, No. 2, 1996. 6.

tracking result, for the development of real-time face tracking[12] J. F. Cohn et al. , “Multimodal Coordination of Facialsystem.

Action, Head Rotation, and Eye Motion during Spontaneous
Smiles”,

Proceedings

of

the

Sixth

IEEE

International

References

Conference on Automatic Face and Gesture Recognition, 2004.

[1] D. Cristinacce and T. Cootes, "Feature Detection and[13] K. Oka and Youichi Sato, “Real-time Modeling of Face

Tracking with Constrained Local Models", Proc. British

Deformation for 3D Head Pose Estimation”, Meeting on Image

Machine Vision Conference, Vol. 3, pp. 929-938, 2006.

Recognition and Understanding 2005(MIRU2005), pp.167-173,[2] K. Hotta et al. , “A Robust Face Detector under Partial

2005.7.

Occlusion”,

IEEE

International

Conference

on

Image

Processing, pp. 597-600, 2004.10.
[3] Y. Matsumoto et al. , ”Development of Real-time Face andGaze Measurement System and Applications to Intelligent
User Interfaces” , Information Processing Society of Japan,
Transactions on Computer Vision and Image Media, Vol. 47,
No. SIG15(CVIM16), pp. 10-21, 2006.10. (in Japanese)[4] Institute for Information Technology, “NRC-IIT Facial