Journal of Robotics, Networking and Artificial Life, Vol. 3, No. 1 (June 2016), 37-40

A System for Facial Expression Analysis of a Person While Using Video Phone
Taro Asada, Yasunari Yoshitomi, Ryota Kato, and Masayoshi Tabuse

Graduate School of Life and Environmental Sciences, Kyoto Prefectural University,
1-5 Nakaragi-cho, Shimogamo, Sakyo-ku, Kyoto 606-8522, Japan
E-mail: {t_asada, r_kato}@mei.kpu.ac.jp, {yoshitomi, tabuse}@kpu.ac.jp
Jin Narumoto

Graduate School of Medical Science, Kyoto Prefectural University of Medicine,Kajii-cho, Kawaramachi-Hirokoji, Kamigyo-ku, Kyoto 602-8566, Japan
E-mail: jnaru@koto.kpu-m.ac.jp

Abstract
We developed a method for analyzing the facial expressions of a person having a conversation on a videophone.
This was implemented on a wireless local area network, consisting of two personal computers and a router; thesubject and the operator are placed in front of each computer. The module contains a function for determining thereference frame and for locally searching for the mouth area in the image to determine the most appropriate
position for each frame.
Keywords: Facial expression analysis, Movement analysis, Mouth area, OpenCV, and Skype.

1. Introduction
In Japan, the average age of the population has been
increasing, and this trend is expected to continue. Alongwith this, the number of older people with dementia
and/or depression living in rural areas is increasing very
rapidly. Due to a mismatch between the number ofpatients and the number of healthcare professionals, it isdifficult to provide adequate psychological assessment
and support for all patients.
To improve the quality of life (QOL) of elderly
people living in a care facility or at home, we havedeveloped a method for analyzing the facial expressionsof a person who is having a conversation with anotherperson on a videophone.1, 2 The video is analyzed using
image-processing software (OpenCV)3 and a previously
proposed feature parameter (facial expression intensity)that is based on the mouth area.1, 2

In the present study, exploiting our reported
researches1, 2, we developed a system for facial
expression analysis of a person while using video phone.
2. Proposed System and Method
2.1. System overview and outline of the method
The platform includes Skype4 for the videophone, andconversations are recorded for the analysis of facial
expressions. In the recorded data, the size of the faces
are standardized, and the data are analyzed by using
OpenCV and the proposed feature parameters for facial
expressions, as outlined below. The Y component
obtained from each frame in the dynamic image is used
for analyzing the facial expressions. The proposed
method consists of (1) the size of the lower part of theface is standardized; (2) the mouth area is extracted; (3)Published by Atlantis Press
Copyright: the authors
37

Taro Asada, Yasunari Yoshitomi, Ryota Kato, Masayoshi Tabuse, Jin Narumoto

the facial expression intensity is measured; (4) thereference frame is selected; (5) the best position formouth area in the frame is determined; (6) utterances
are evaluated; and (7) the feature parameter for facial
expression strength is calculated. In the following
subsections, (2), (3), (4), (5) and (7) are explained indetail. On the details for (1) and (6), see Refs 1-2.
The structure of the proposed system is shown in
Fig.1. We developed a wireless local area network,
consisting of two personal computers (PCs) and one
router, in which the subject is placed in front of a PC
with a module for analyzing facial expressions, and thesubject converses via videophone with the module
operator, who is placed in front of the other PC.

As the feature parameters for expressing facialexpression, we selected 15 low-frequency components
from the 2D-DCT coefficients; this did not include thedirect current component. Next, we obtained the meanof the absolute value of each of these components in thearea of the mouth. In total, we obtained 15 values as
elements of the feature vector. The facial expression
intensity is defined as the norm of the difference vector,
which is a vector of the difference between the feature
vector for the neutral facial expression and that for theobserved expression, and it can be used to analyzechanges in facial expression.
The candidate of facial expression intensity definedas the norm of the difference vector between two feature
vectors is used for selection of the reference frame as
described in the section 2.4.
2.4. Selection of the reference frameFig. 1. Structure of the proposed system.

2.2. Extraction of the mouth area
Next, by using OpenCV, the mouth area is extracted asa rectangular shape.1 The mouth area is selected because
it is where the difference between neutral and happy
facial expressions is most distinct. An example of a face
image and the extracted image of the mouth area isshown in Fig. 2.

We propose a method for automatically selecting thereference frame to be used for measuring the facial
expression intensity; this can be used for a moving
image in which the initial facial expression is neutral
and there is no utterance. The existence of an utterance
is determined by a method that we proposed in aprevious paper.1, 2 The frame prior to any utterance isused as the target in following procedure.
We calculated the sum of the norms of thedifference vectors for the feature vector calculated foreach of the candidates for the reference frame, and wedesignated the one for which this sum was minimized as
the reference frame. The facial expression intensity isthen obtained as the norm of the vector of thedifferences between the feature vector and the featurevector of the reference frame.
2.5. Correction of position of mouth area

Fig. 2. Whole-face image (left), extracted image of the moutharea (right).

2.3. Measurement of facial expression intensity
For the Y component of the selected frame, the feature
vector for the facial expression is extracted for themouth area; this is performed by using a 2D-DCT foreach section of 8×8 pixels.

There is no guarantee that the center of the extracted
mouth area will remain in a suitable and constantposition by eyes, and in fact, we observed that it didvary. This change in position could affect the evaluatedfacial expression intensity. Therefore, we propose a way
to decrease the influence on the facial expression
intensity due to changes in the location of the moutharea.
After using OpenCV to extract the mouth area, thebest position of the mouth area under the condition on
the shift of the area is found by deciding each shiftvalue for the area in the range of -3 to 3 pixels in each

Published by Atlantis Press
Copyright: the authors
38

A System for Facialof the horizontal and vertical directions, in order to givethe smallest value of facial expression intensity for theframe.

In diagnosing a patient having dementia and/ordepression, it might be useful for healthcare
professionals to evaluate the strength of facial
expressions by using a simple measure.1 Moreover, it
might be more advantageous for a diagnosis of dementia
and/or depression to separately evaluate the strength of
the facial expression as a speaker and a listener.1
Therefore, we measure the feature parameter for facial
expression strength as the average of facial expression
intensity as a speaker and a listener1, if necessary.
3. Experiments
3.1. Experimental environment
The experiment was performed in the following
computational environment: the PC set in front ofsubject A was a Dell Vostro 3350; CPU: Intel Core i72620M 2.7 GHz; 4.0 GB memory; OS: MicrosoftWindows 7 Professional. The development language
was Microsoft Visual C++ 2008 Express Edition.
Three males (subject A in his 30s, subject B in his20s, and subject C in his 40s; C is a psychiatrist)
participated in addition to an operator. Each experimentconsisted of a two-way videophone conversation thatlasted from 27 to 36 seconds. Experiment 1 was aconversation between subject A in Tokyo and subject B
in Kyoto, and it was conducted using Skype.
Experiment 2 took place face-to-face as an interview tosubject A by subject C. We saved the visual and audio
information as AVI files, and these were then used formeasuring the feature parameters of the facial
expressions. The size of the image frame was 640×480
pixels, and the size of the standardized lower part of theface image was set to 240×96 pixels.
3.2. Results and discussion
Fig. 3 shows the mouth area of subject A at thebeginning of each experiment.

Fig. 3. The mouth area of subject A at the beginning ofExperiment 1 (left), and at the beginning of Experiment 2(right).

The facial expression intensity for subject A wasrecorded during both experiments. In Experiment 1, thetiming of utterances was 11.4 seconds. During his
utterances in Experiment 1, there were four local peaksin the facial expression intensity of subject A. Theseoccurred at approximately 3, 6, 15, and 21 seconds after
the starting point (see Fig. 4). During the times whensubject A was not making utterances during Experiment
1, there were five local peaks in his facial expression
intensity; these were at approximately 3, 7, 11, 13, and23 seconds after the starting point (see Fig. 5);characteristic images of the face and mouth area areshown in Fig.5. In Experiment 2, the timing ofutterances was 12.4 seconds; there were four local peaksin the facial expression intensity for subject A; thesewere at approximately 14, 16, 23, and 26 seconds after
the starting point (see Fig. 6). In both Experiments 1 and2, the images of the face and mouth areas at thecharacteristic timing points show that the proposed
method can quantitatively determine the facialexpression (see Figs. 5 and 6).
The feature parameters for the facial expression
intensity are shown in Table 1. Note that the intensity
tended to be greater during face-to-face conversation
than during a videophone conversation.
Facial expression intensity

2.6. Feature parameter for facial expression
strength3530252015
1050051015Time (s)2025Fig. 4. Changes in the facial expression intensity for subject A
at times of utterance during Experiment 1.

Published by Atlantis Press
Copyright: the authors3935

Facial expression intensity

Facial expression intensity

Taro Asada, Yasunari Yoshitomi, Ryota Kato, Masayoshi Tabuse, Jin Narumoto

3025A2A12015
1050051015Time (s)A1202535

A330

A425
20
15
10
5005A3A2

1015Time (s)202530

A4Fig. 5. Changes in the facial expression intensity for subject Aat times of no utterance during Experiment 1 (upper graph).
Whole-face images and mouth images are shown for twomoments, A1 and A2, which are indicated on the graph (lowerimages).

Fig. 6. Changes in the facial expression intensity for subject A
during Experiment 2 (upper graph). Whole-face images andmouth images are shown for two moments, A3 and A4, which
are indicated on the graph (lower images).

The processing time for analyzing the facial
expressions using the proposed system was 126 s forExperiment 1 and 135 s for Experiment 2.

Acknowledgements

Table 1. Feature parameter for facial expressions of subject
A.
Experiment
12

Utterance
withwithoutwith & without

Feature parameter

5.65
6.96
10.67

4. Conclusion
We developed a wireless local area network system,
consisting of two PCs and one router, which includes amodule for analyzing the facial expressions of a subject
engaged in a two-way videophone conversation.
The results show the usefulness of the proposed
method. As an area of future work, we intend to develop
a related method for estimating the mental state and/or
recognition ability of a patient.

This research was supported by COI STREAM of the
Ministry of Education, Culture, Sports, Science, andTechnology of Japan.
References
1. T. Asada, Y. Yoshitomi, R. Kato, M. Tabuse, and J.
Narumoto, Quantitative evaluation of facial expressionsand movements of persons while using video phone, J.
Robotics, Networking and Artif. Life 2(2) (2015) 111114.
2. T. Asada, Y. Yoshitomi, R. Kato, M. Tabuse and J.
Narumoto, Analysis of facial expressions robust against
small imperfections in mouth-part area extraction fromface-images of persons while using video phone (inJapanese), in Proc. of Human Interface Symposium 2015
(Japan, Hakodate, 2015), pp.187-190.
3. Open CV. http://opencv.org/ Accessed 1 December 2015.
4. Skype Web page, http://www.skype.com/ Accessed 5November 2015.

Published by Atlantis PressCopyright: the authors
40