Journal of Robotics, Networking and Artificial Life, Vol. 2, No. 2 (September 2015), 111-114

Quantitative Evaluation of Facial Expressions and Movements of Persons While Using Video Phone
Taro Asada, Yasunari Yoshitomi, Ryota Kato, and Masayoshi Tabuse

Graduate School of Life and Environmental Sciences, Kyoto Prefectural University,
1-5 Nakaragi-cho, Shimogamo, Sakyo-ku, Kyoto 606-8522, Japan
E-mail: {t_asada, r_kato}@mei.kpu.ac.jp, {yoshitomi, tabuse}@kpu.ac.jp
Jin Narumoto

Graduate School of Medical Science, Kyoto Prefectural University of Medicine,Kajii-cho, Kawaramachi-Hirokoji, Kamigyo-ku, Kyoto 602-8566, Japanjnaru@koto.kpu-m.ac.jp

Abstract
A video is analyzed by image processing and the feature parameters of facial expressions and movements, which
are extracted in the mouth area. The feature parameter for expressing facial expressions is defined as the average ofthe facial expression intensity. The parameter for expressing movement of a person is defined as the average of theabsolute value of the vertical coordinate for the center of gravity of the mouth area in the relative coordinate
system. The experimental result shows the usefulness of the proposed method.
Keywords: Facial expression analysis, Movement analysis, Mouth area, OpenCV, and Skype.

1. Introduction
In Japan, the average age of the population has been
increasing, and this trend is expected to continue.
Consequently, the number of older people with dementia
and/or depression living in rural areas is increasing very
rapidly. Due to the mismatch between the number ofpatients and healthcare professionals, it is difficult toprovide psychological assessment and support for therural patients.
To improve the quality of life (QOL) of elderly
people living in a care facility or at home, we have beendeveloping a method for analyzing the facial
expressions of a person while speaking with anotherperson on a video phone.1-3 In the present study, wedeveloped a method for analyzing facial expressions
and movements of a person while speaking with anotherperson by using our previously reported method2, whichis standardization of the size of face to be analyzed, andnewly proposed feature parameters of facial expressions
and movements of a person.
2. Proposed Method
2.1. System overview and outline of the method
The platform includes Skype4 for the video phone. To
record the audio and video dialogue, Netralia Pty Ltd’s
VodBurner5 and Tapur6 are introduced. Conversations
are recorded for the analysis of facial expressions andmovements of a person. The recorded data are analyzed
by using the image processing software Open Source
Computer Vision Library for real-time computer visiondeveloped by Intel (Open CV)7, standardization of thesize of face to be analyzed, and the newly proposed

Published by Atlantis Press
Copyright: the authors
111

Taro Asada, Yasunari Yoshitomi, Ryota Kato, Masayoshi Tabuse, Jin Narumotofeature parameters of facial expressions and movementsof a person, as described in the following subsections.
The Y component obtained from each frame in thedynamic image is used for analyzing facial expressions
and movements of a person. The proposed method
consists of (1) standardization of the size of the lower
part of the face area, (2) extraction of the mouth area,(3) measurement of facial expression intensity, (4)judgment of utterance, (5) calculation of the featureparameter for facial expression strength, and (6)calculation of the feature parameter for movements of aperson. In the following subsections, these six areexplained in detail.
2.2. Standardization of lower part of face area insize
First, the face area obtained from each frame in thedynamic image is extracted by using the classifier for afront-view face included in OpenCV. In the classifier, theHaar-like feature parameter and the Adaboost algorithm
for learning are used.8 It is assumed that the distance
between a subject and the camera is almost always thesame during a conversation by using Skype. The face ofthe frame, where the face area is extracted using OpenCV
and has a minimum size among those for a period in thedynamic imaging, is assumed to be the most likely frontview among those in the period1, and that a frame isselected and used. In the present study, we set 1/3
seconds as the period. Then, the lower part of the facearea extracted by the above method is standardized inlength and width for extracting the mouth area. This
standardization in size is performed with the aim of not
only improving the performance of extracting the mouth
area by OpenCV but also normalizing the featureparameter generation by using 2D-DCT (Discrete CosineTransform) at the mouth area. Under the circumstancethat the size of the face in a dynamic image cannot bekept constant every time, this standardization in size isindispensable for reliably measuring the featureparameters described in Sections 2.6 and 2.7.
2.3. Extraction of mouth area
Next, by using OpenCV, the mouth area is extracted as arectangular shape. The mouth area is selected because thedifference between the facial expressions of neutral andhappy distinctly appears there. Fig. 1 shows an example
of a face image: the lower part of the face image before

and after standardization of size, and the image of theextracted mouth area.

Fig. 1. Total face image (upper), lower part of face image
before (lower left) and after (lower center) standardization ofsize, and mouth-area image extracted (lower right).

2.4. Measurement of facial expression intensity
For the Y component of the frame selected by theprocessing described above, the feature vector of facial
expression is extracted in the mouth area by the use of
2D-DCT performed for each domain having 8×8 pixels.
We select 15 low-frequency components of the2D-DCT coefficients, except for the direct current
component, as the feature parameters for expressing
facial expression.2 Then, we obtain the mean of theabsolute value for each 2D-DCT coefficient component
in the area of the mouth.2 In total, we obtain 15 values
as elements of the feature vector. The facial expression
intensity, defined as the norm of the difference vector
between the feature vector of the neutral facial
expression and that of the observed expression, can beused for analyzing the change of facial expression.22.5. Judgment of utterance
The sound data are smoothed and sampled to erase noise.
Then,
all
sampleddata
thatfallwithinxs  14 s , xs  14 s  , where xs and  s express theaverage and the standard deviation, respectively, of thesound data value for one second under the condition of noutterance, are considered to be in the range of noutterance. When at least one sampled datum has a value
outside xs  14 s , xs  14 s  , our system judges thatthe sound data contain an utterance after erasing the noise.

Published by Atlantis Press
Copyright: the authors
112Quantitative Evaluation of Facial Expressions

3.2. Results and discussion

2.6. Feature parameter for facial expression
strength
In diagnosing a patient having dementia and/ordepression, it might be useful for healthcare
professionals to evaluate the strength of facial
expressions by using a simple measure. Moreover, it
might be more advantageous for a diagnosis of dementia
and/or depression to separately evaluate the strength of
the facial expression as a speaker and a listener.
Therefore, we measure the feature parameter for facial
expression strength as the average of facial expression
intensity in the four cases of (1) both subjects A and B
speak, (2) subject A speaks and subject B does notspeak, (3) subject A does not speak and subject B
speaks, and (4) both subjects A and B do not speak, byusing the method for judging an utterance described inSection 2.5.

Fig. 2 shows the mouth-area images at the starting point.
The facial expression intensity changes of subjects Aand B during their conversation (Fig. 3) and the changesof coordinates of the center of gravity on the mouth area
(Fig. 4) are shown. In Fig. 5, face images are shown at
the characteristic timing positions. Feature parametersof facial expressions and movements of the subjects areshown in Table 1. The definitions of these parameters
are described in Sections 2.6 and 2.7.
As shown in Figs. 3 and 5, facial expression intensity
was very sensitive for facial expression change. Bothsubjects A and B did not move vertically so much during
conversation (Fig. 4 and Table 1). The number of moutharea images extracted by OpenCV was increased for subject
A from 196 to 197 by performing size standardization

2.7. Feature parameter for movements of a personHead movements such as a nodding during conversation
might suggest a mental state and/or recognition abilityof a patient having dementia and/or depression.
Therefore, as the feature parameter for movements of aperson, we measure the average of the absolute value ofthe vertical coordinate for the center of gravity of themouth area in the relative coordinate system in the fourcases described in Section 2.6. The relative coordinate
system is defined by using the mouth area extracted at
the starting point for measuring the feature parameterfor the movements of a person. At the starting point, theheight of the mouth area is set to be one and the verticalcoordinate for the center of gravity of the mouth area isset to be 0 in the relative coordinate system.

Fig. 2. Mouth-area images of subjects A (left two images) andB (right two images) at the starting point with (left side) andwithout (right side) size standardization before extracting themouth area.

3. Experiment3.1. Condition
Two males (subject A in his 50s and subject B in his20s) participated in an experiment where they had aconversation for about 70 seconds by using Skype. The
videos saved by VodBurner were transformed into AVI
files, and the audios saved by Tapur were transformed
into WAV files. The AVI files were used for measuringfeature parameters of facial expressions and movementsof the subject. The WAV files were used for judgmentof an utterance. The size of the image frame was
720×480 pixels, and the size of the standardized lower
part of a face image was set to 240×96 pixels.

Fig. 3. Facial expression intensity change of subjects A and B
during the conversation between the two subjects with size
standardization before extracting the mouth area.

Fig. 4. Changes of coordinates of center of gravity on moutharea with size standardization before extracting mouth area.

Published by Atlantis Press
Copyright: the authors
113

Taro Asada, Yasunari Yoshitomi, Ryota Kato, Masayoshi Tabuse, Jin Narumoto(Table 1). The value of the feature parameter for facial
expression was relatively high for subject A in all threecases, whereas it was relatively high for subject B in theonly case that subject B spoke and subject A did not
speak (Table 1). Because the period in which bothsubjects A and B spoke was very short, the data are notdescribed in Table 1.
4. Conclusion

Fig. 5. Face images at characteristic timing positions for facial
expression intensity value of subject A; upper: 0 (startingpoint), lower left: maximum, lower right: minimum exceptthat at starting point.
Table 1. Feature parameters for facial expressions andmovements.
(1) With size standardization before extracting mouth area
Utterance
Feature parameter
Subject
FacialMovementA
B
expression of person
without
without
11.41
0.05
AB

withwithout

9.47

0.08

without

with10.50

0.05

BA

without

without

4.530.05

withwithout

12.51

0.07

without
with
3.14
0.06
(2) Without size standardization before extracting mouth area
Utterance
Feature parameter
Subject
FacialMovementAB
expression of person
without
without
8.55
0.03
AB

with

without

8.50

0.04

without

with

10.06

0.05

BA

without

without

5.48

0.04

with

without

10.88

0.05

without

with

5.18

0.03

before extracting the mouth area, while it was increased
for subject B from 27 to 138 by standardization. Thoughmouth-area images were influenced by size
standardization before extracting the mouth area (Fig. 2),
the feature parameter values of both facial expressions
and movements of subjects were not influenced so muchby size standardization before extracting the mouth area

A video is analyzed by image processing and the newly
proposed feature parameters of facial expressions andmovements. The experimental result shows theusefulness of the proposed method. In future work, wewill develop the method for estimating the mental state
and/or recognition ability of a patient by using theproposed method.
Acknowledgment
This research is partially supported by COI STREAM ofthe Ministry of Education, Culture, Sports, Science andTechnology of Japan.
References1. T. Asada, Y. Yoshitomi, A. Tsuji, R. Kato, M. Tabuse, N.
Kuwahara, and J. Narumoto, Method of facial expression
analysis of person while using video phone, in Proc. of Human
Interface Symposium 2013 (Japan, Tokyo, 2013), pp.493-496.
2. T. Asada, Y. Yoshitomi, A. Tsuji, R. Kato, M. Tabuse, N.
Kuwahara, and J. Narumoto, Facial expression analysis
while using video phone, in Proc. of Int. Conf. on Artif.
Life and Robotics (Japan, Oita, 2014), pp.230-234.
3. J. Narumoto, N. Kuwahara, Y. Yoshitomi, T. Asada, Y.
Kato, H. Kamimura, K. Fukui, Development of support
system for patients with dementia through teleconference
system, in Proc. of 4th World Conf. of Asian Psychiatry(Thailand, Bangkok, 2014), pp.1-4.
4. Skype Web page, http://www.skype.com/ Accessed 5
November 2013.
5. VodBurner Web page, http://www.vodburner.com/
Accessed 11 July 2014.
6. Tapur Web page, http://www.tapur.com/jp/ Accessed 18
December 2014.
7. OpenCV Web page, http://opencv.willowgarage.com/
Accessed 11 July 2014.
8. P. Viola and M. Jones, Rapid object detection using aboosted cascade of simple features, in Proc. of the 2001
IEEE Computer Society Conf. on Computer Vision andPattern Recognition (USA, Kauai, 2001), Vol.1, pp.511-518.

Published by Atlantis Press
Copyright: the authors
114