Journal of Robotics, Networking and Artificial Life, Vol. 2, No. 4 (March 2016), 258-262

Facial Expression Analysis while Using Video Phone
Taro Asada1, Yasunari Yoshitomi1, Airi Tsuji2, Ryota Kato1, Masayoshi Tabuse1, Noriaki Kuwahara2, and Jin Narumoto3

1: Graduate School of Life and Environmental Sciences Kyoto Prefectural University, 1-5 Nakaragi-cho, Shimogamo,Sakyo-ku, Kyoto 606-8522, Japan, {t_asada, r_kato}@mei.kpu.ac.jp, {yoshitomi, tabuse}@kpu.ac.jp,
2: Graduate School of Science and Technology, Kyoto Institute of Technology,Matsugasaki, Sakyo-ku, Kyoto 606-8585, Japan, diff.dim0505@gmail.com, nkuwahar@kit.ac.jp
3: Graduate School of Medical Science, Kyoto Prefectural University of Medicine,Kajii-cho, Kawaramachi-Hirokoji, Kamigyo-ku, Kyoto 602-8566, Japan, jnaru@koto.kpu-m.ac.jp

Abstract
We have been developing a method for analyzing the facial expressions of a person using a video phone system(Skype) to talk to another person. The video is recorded and analyzed by image processing software (OpenCV) andthe newly proposed feature vector of facial expression. The newly proposed facial expression intensity can be usedto analyze a change of facial expression. The judgment of speaking is performed in our study. The experimental
results show the usefulness of the proposed method.
Keywords: Facial expression analysis, Mouth area extraction, OpenCV, Skype, Video phone.

1. Introduction
In Japan, the average age of the population has been
increasing, and this trend is expected to continue.
Because of this trend, the number of elderly people withdementia and/or depression, especially those living inrural areas, is increasing very rapidly. Due to themismatch between the number of patients and thenumber of healthcare professionals, it is difficult toprovide adequate psychological assessments andsupport for all patients.
According to the experiences of psychiatrists in the
medical treatment, people with dementia and/ordepression have a poor change of facial expressions. On
the other hands, the number of elderly people withdementia and/or depression is increasing rapidly
because of the recent growing trend of aging. Therefore,
a method for assessing emotional state of an elderly

person with dementia and/or depression by analyzing
facial expressions is missing.
Information and communication technology is apromising method for overcoming the difficulty causedby the lack of adequate healthcare. In Japan, the first
inexpensive connection to the Internet became available
only recently in rural areas and high-quality freesoftware, such as Skype1, is being distributed.
To improve the quality of life of elderly people
living in a home or a healthcare facility, we propose amethod for analyzing the facial expressions of a personusing a video phone system (Skype) to speak withanother person. In the present study, the phone video isrecorded and analyzed by image processing software(OpenCV) and the newly proposed feature vector offacial expression, which is extracted in the mouth area.
Moreover, the judgment of speaking is performed byusing the intensity of the sound wave.

Published by Atlantis Press
Copyright: the authors
258

Taro Asada, Yasunari Yoshitomi, Airi Tsuji, Ryota Kato, Masayoshi Tabuse, Noriaki Kuwahara, Jin Narumoto

2.3. Measurement of facial expression intensity

2. Proposed Method
2.1. System overview and outline of the method
As already mentioned, the video phone is Skype.1
VodBurner (Netralia Pty Ltd.)2 is introduced forrecording the audio and video dialogue. Tapur3 is also
introduced for recording the audio data. Conversations
are recorded for the analysis of facial expression. The
recorded data are analyzed by image processing
software, Open Source Computer Vision Library (OpenCV, Intel), for real-time computer vision4 and the newly
proposed feature vector of facial expression described inthis paper. The Y component obtained from each framein the dynamic image is used for measuring the facial
expression intensity. The proposed method consists of(1) extraction of the mouth area, (2) measurement offacial expression intensity, and (3) judgment ofutterance. In the following subsections, these three are
explained in detail.
2.2. Extraction of mouth area from a dynamic image
First, the face area is extracted from each frame in thedynamic image by the classifiers for a front-view face
included in OpenCV. The Haar-like feature parametersand Adaboost algorithm for learning are used as theclassifiers.5 It is assumed that the distance between asubject and the camera generally remains constantduring a Skype conversation. However, we observedthat the size of the face area tends to increase when theface deviates from a front-view image. In this case, it isdifficult for OpenCV to extract the face area. Therefore,
the minimum size of dynamic image within a specified
period is assumed to be the most likely front view. In
the present study, we set one second as the period. Next,
by using OpenCV, the mouth area is extracted for theframe selected by the face-area size criterion describedabove. The mouth area is selected because thedifference between the facial expressions of neutral andhappy distinctly appears in this area.
It is essential for assessing emotional state of aperson with dementia and/or depression to judge
whether he or she can react with a facial expression of“happy” during the talk. Therefore, we think that themouth area is enough to be analyzed at the present stageof our study.

For the Y component of the frame selected by theprocessing described above, the newly proposed feature
vector of facial expression is extracted in the mouth area
by applying 2-dimensional Discrete Cosine Transform(2D-DCT) for each domain of 8 × 8 pixels.
The mouth area used for measuring the facial
expression intensity has n blocks of 8×8 pixels, wheren is obtained as n = [a / 8] × [b / 8] , a and b denote thenumber of pixels of the mouth area in the face area
obtained by OpenCV in the vertical and horizontal
directions, respectively, and [x] equals the maximum
integer that does not exceed x .
The high-frequency components of the 2D-DCT
coefficients tend to express a minute change in the data,
and thus result in the presence of noise. Therefore, we
select 15 low-frequency components of the 2D-DCT
coefficients, except for a direct current component, as
the feature parameters for expressing facial expression(Fig. 1). This selection of 2D-DCT coefficients is
popular among researchers in facial expression
recognition.6

Fig. 1. Special frequency bands used for the analysis.

Because we do not know the combination of thespecific face location and the frequency component of
the 2D-DCT coefficients to successfully recognize afacial expression, we adopt the strategy described
below.
To gather useful information from the mouth area,
we obtain the absolute value of 2D-DCT coefficients,
then we obtain the mean of the absolute value for each
2D-DCT coefficient component in the mouth area (Fig.
2). The number of 2D-DCT coefficient components is
15. Therefore, we obtain 15 values as the elements ofthe feature vector. The facial expression intensity,
defined as the norm of the difference vector between thefeature vector of the neutral facial expression and that of

Published by Atlantis Press
Copyright: the authors259

Facial Expression Analysis whileconversation for approximately 80 seconds. The videos
saved by VodBurner were transformed into AVI files,
and WAV files were saved by Tapur. The AVI files
were used for measuring the facial expression intensity.
The WAV files were used for judgment of an utterance.
3.2. Results and discussion

Fig. 2. Schematic diagram of the DCT feature parameter
calculation in the mouth area.7

the observed expression, can be used for analyzing achange of facial expression.
2.4. Judgment of utterance
Combining the video signal obtained from Skype withthe sound signal, we can distinguish the facialexpression with speaking from that without speaking.
Based on the method reported in Refs. 8–10, the sounddata are smoothed and sampled to erase noise. The
judgment of speaking is performed by using a thresholdof the sound intensity. The threshold is determined by
the average and the standard deviation of the soundintensity when the subject does not speak in the soundenvironment where Skype is used. The thresholds forthe sound data values are set as xs − 14s s and xs + 14s s ,where x s and s s express the average and the standard
deviation, respectively, of the sound data value for one
second under the condition of no utterance.
Then, every sampled data that falls within

Facial expression intensity changes of subjects A and B
during their conversation were recorded (Fig. 3). The
timing of utterances (Fig. 4), and the timing of noutterances (Fig. 5) are shown. In both Figs. 4 and 5, face
images and mouth images show the characteristic
timing positions for the facial expression intensity.
Subject A expressed four local peaks of facial
expression intensity at approximately 20, 37, 47, and 71
seconds from the start point, while subject B expressed six
local peaks of facial expression intensity at approximately
40, 45, 55, 65, 71, and 75 seconds from the start point (Fig.
3). For the timing of utterances during their conversation,subject A expressed four local peaks of facial expression
intensity at approximately 18, 37, 47, and 71 seconds
from the start point, while subject B expressed five local
peaks of facial expression intensity at approximately 45,55, 65, 71, and 75 seconds from the start point (Fig. 4).
For the timing of no utterances during their conversation,subject A expressed three local peaks of facial expressionintensity at approximately 20, 46, and 70 seconds fromthe start point, while subject B expressed one local peakof facial expression intensity at approximately 40 seconds
from the start point (Fig. 5).

[xs − 14s s , xs + 14s s ] are considered to be the range ofno utterance. When at least one sampled datum has avalue outside [x s − 14s s , x s + 14s s ] , our system judgesthat the sound data contain an utterance.
3. Experiment3.1. Condition
Two males (subjects A and B) in their 20s participatedin the experiment. Using Skype, the two subjects held aFig. 3. Facial expression intensity change of subjects A and B
during their conversation.

Published by Atlantis Press
Copyright: the authors
260

Taro Asada, Yasunari Yoshitomi, Airi Tsuji, Ryota Kato, Masayoshi Tabuse, Noriaki Kuwahara, Jin Narumoto

a1

A1

a2

A2

a3

A3b1

B1b2

B2b3

B3Fig. 4. Facial expression intensity changes (upper graph), face
images (lower left side), and mouth image (lower right side) ofsubjects A and B at the timing of utterances during theirconversation.

Fig. 5. Facial expression intensity changes (upper graph), faceimages (lower left side), and mouth images (lower right side)of subjects A and B at the timing of no utterances during theirconversation.

Published by Atlantis Press
Copyright: the authors
261

Facial Expression Analysis while

Subject B mainly made an utterance during thesecond half of the conversation (Fig. 4), while the facial
expression intensity increased gradually for the timingof no utterances during the first half of the conversation(Fig. 5). Subject A expressed two local peaks of facial
expression intensity at approximately 18 and 37 seconds
from the start point, and subject B expressed no local
peaks of facial expression intensity when the data werelimited to the timing of utterances during the first half ofthe conversation (Fig. 4). Just after the only local peakof facial expression intensity having no utterance atapproximately 40 seconds from the start point, subject B
expressed continual peaks of facial expression intensitywith utterances.
The images of the face and mouth areas at thecharacteristic timing points show that the proposed
method can quantitatively express the facial expression(Figs. 4 and 5).
4. Conclusion
We proposed a method for analyzing the facial
expressions of a person while speaking with a video
phone system (Skype). The recorded video is analyzed
by image processing software (OpenCV) and the newly
proposed feature vector of facial expression, which isextracted in the mouth area by applying 2D-DCT. The
facial expression intensity, defined as the norm of thedifference vector between the feature vector of theneutral facial expression and that of the observed
expression, can be used to analyze the change of facial
expression. The judgment of speaking is performed byusing the intensity of the sound wave. The experimental
results show the usefulness of the proposed method.

3. Tapur Web page, http://www.tapur.com/jp/ Accessed 1December 2013.
4. OpenCV Web page, http://opencv.willowgarage.com/Accessed 1 December 2013.
5. P. Viola and M. Jones, Rapid object detection using aboosted cascade of simple features, in Proc. of the 2001
IEEE Computer Society Conf. on Computer Vision andPattern Recognition (USA, Kauai, 2001), Vol.1, pp.511518.
6. T. Sakaguchi and S. Morishige, Real-time facial
expression recognition based on the 2-dimensional DCT,
Trans IEICE J80-D-Ⅱ(6) (1997) 1547-1554.
7. Y. Yoshitomi, M. Tabuse, and T. Asada, Facial
expression recognition using thermal image processing,in Image processing: methods, applications andchallenges ed. V. H. Carvalho (Nova Science Publisher,
New York, 2012), pp. 57–85.
8. F. Ikezoe, M. Nakano, Y. Yoshitomi, and M. Tabuse,
Facial expression recognition using thermal face image
automatically acquired in speaking (in Japanese), in Proc.
Human Interface Symp. 2005, (Japan, Fujisawa, 2005),
pp. 7–12.
9. M. Nakano, Y. Yoshitomi, and M. Tabuse, Efficientfacial expression recognition using thermal face image inspeaking and its application to analysis of individualvariations (in Japanese), in Proc. of Human Interface
Symp. 2006, (Japan, Kurashiki, 2006), pp. 1151–1156.
10. M. Nakano, Robust facial expression recognition forvarious speakers (in Japanese), Master’s thesis, Dept. ofEnvironmental Information Graduate School of Human
Environmental Sciences Kyoto Prefectural University,(2008), pp 1-70.

Acknowledgements
We would like to thank the subjects who participated inthe experiments. This research is supported by SCOPE(122307003) of the Ministry of Internal Affairs ofCommunications of Japan and COI STREAM of theMinistry of Education, Culture, Sports, Science andTechnology of Japan.
References
1. Skype Web page, http://www.skype.com/ Accessed 5
November 2013.
2. VodBurner Web page, http://www.vodburner.com/
Accessed 1 December 2013.

Published by Atlantis Press
Copyright: the authors
262