Journal of Robotics, Networking and Artificial Life, Vol. 3, No. 3 (December 2016), 148-151

Emotion Recognition of a Speaker Using Facial Expression Intensity of Thermal Image andUtterance TimeYuuki Oka

NTT DATA Financial Solutions Corp.
Kandanishikicho, Chiyoda-ku, Tokyo 101-0054, Japan
Yasunari Yoshitomi, Taro Asada, and Masayoshi Tabuse

Graduate School of Life and Environmental Sciences, Kyoto Prefectural University,
1-5 Nakaragi-cho, Shimogamo, Sakyo-ku, Kyoto 606-8522, Japan
E-mail: {yoshitomi, tabuse}@kpu.ac.jp, t_asada@mei.kpu.ac.jp}
http://www2.kpu.ac.jp/ningen/infsys/English_index.html

Abstract
Herein, we propose a method for recognizing human emotions that utilizes the standardized mean value of facial
expression intensity obtained from a thermal image and the standardized mean value of the time at utterance. In thisstudy, the emotions of one subject could be discerned with 76.5% accuracy when speaking 23 kinds of utterances
while intentionally displaying the five emotions of “anger,” “happiness,” “neutrality,” “sadness,” and “surprise.”
Keywords: Emotion recognition, Mouth and jaw area, Thermal image, Utterance judgment.

1. Introduction
This research is intended to facilitate the developmentof robots that can perceive human feelings and mentalstates. While various mechanisms for recognizing
human feelings from facial expressions have received
considerable attention in the field of computer visionresearch in recent years, the results achieved to date fall
far short of human capabilities. This is due to thelimited accuracy of facial expression recognition, whichis influenced by inevitable gray level changes that result
from local variations in lighting, shade, reflection, anddarkness.
To avoid this problem and to facilitate thedevelopment of a robust facial expression recognition
method that would be applicable to widely varied
lighting conditions, we use imagery obtained from
infrared rays to describe the thermal distribution of a

subject’s face.1-4 The timing used by a robot whenattempting to recognize facial expressions is also
important because the required processing can be timeconsuming. Accordingly, we adopted utterances as the
key for expressing human feelings because humans tendto speak aloud when expressing their feelings.2-4
In the present study, using 23 different utterance
combinations based on the first and last vowels in eachutterance, we investigated the performance of ourpreviously proposed method5 of recognizing emotions
by utilizing facial expression intensity6,7 and theutterance time.
2. Evaluation Method
The proposed method consists of (1) mouth and jaw area
extraction, (2) facial expression intensity measurement, (3)utterance judgment, and (4) calculation of featureparameters for facial expression and voice.5

Published by Atlantis Press
Copyright: the authors
148

Yuuki Oka, Yasunari Yoshitomi, Taro Asada, Masayoshi Tabuse

2.1. Extraction of mouth and jaw areas
Six facial areas extracted at 0.1 frames per second via thethermal image processing reported in our previous study3
are used in the processing of dynamic thermal imagery, as
shown in Fig. 1. The mouth and jaw area was selected
because facial expression differences between neutrality
and happiness are most distinctive in this area.6 Fig. 2
shows thermal image examples of whole face and themouth and jaw areas.

Fig. 1. Blocks for extracting partial face areas (left), and thethermal image after partial face extraction (right).2Fig. 2. Thermal face (left) and mouth and jaw area imagery(right).6

2.2. Measurement of facial expression intensity
For the extracted frames, the expression feature vectors in
the mouth and jaw area are extracted by applying a twodimensional discrete cosine transform (2D-DCT) for each8 × 8 pixel domain.6 To accomplish this, we begin by
selecting 15 low-frequency components of the 2D-DCT
coefficients, excluding the direct current component, as thefacial expression feature parameters.6
Next, we obtain the mean of the absolute value foreach 2D-DCT coefficient component in the mouth and jaw
area.6 A total of 15 values are obtained and used as feature
vector elements. Facial expression intensity, which isdefined as the norm of the differences between the neutral
facial expression feature and the observed expression
vectors, can then be used for analyzing facial expression
changes.6
2.3. Utterance judgment
After sample collection, the sound data are smoothed toerase noise. Then, all sampled data that fall within[xs − 14s s , xs + 14s s ] , where xs and s s express theaverage and the standard deviation of the sound data
value, respectively, for one second under the noutterance condition, are considered to be the range forno utterance.6 When at least one sampled datum has avalue outside [xs − 14s s , xs + 14s s ] , our system judgesthat the sound data contains an utterance.6
2.4. Feature parameters
In our proposed method,5 two parameters are used asfeature vector elements. One is the mean of thestandardized facial expression intensity at the time of eachutterance and for 0.3 s before and after the utterance, while
the other is the standardized time at each utterance. The
standardization used for creating feature parameters isexpressed by Eq. (1).
xi , j − xi ,(1)xi*, j =σiwhere xi*, j , x i, j , xi , and σ i express the standardized
feature parameter, the measured feature parameter, theaverage, and the standard deviations of the measured
feature parameters of the training data, respectively, andi, j denote the respective number (1 or 2) of the featureparameter and respective number ( 1, 2,  , m ) of theutterance, respectively. Then, using the Ward method,clustering is performed separately for the training andtest data to determine major and minor clusters for eachclass of the emotions in the feature vector space. Next,
to recognize emotions from the test data, the center-ofgravity coordinates of the major cluster for eachemotion class of the training and test data is used.
3. Experiments
3.1. Conditions
Thermal imagery produced by a TVS-700 thermal video
system (Nippon Avionics, Tokyo, Japan) and sounds
captured by an ECM-23F5 Electret condenser
microphone (Sony, Tokyo, Japan), as amplified by an
AT-PMX5P mixer (Audio-Technica, Tokyo, Japan),were transformed into a digital signal via an ADVC-300
analog/digital (A/D) converter (Thomson Canopus,Kobe, Japan).
The converted signals were input into an Optiplex
780 personal computer (PC) (DELL, Round Rock, TX)
equipped with an E8400 3.00 GHz Core 2 Duo central
processing unit (CPU) (Intel, Santa Clara, CA) and 3.21
GB of main memory, as well as a 1394-PCI3/DV6
IEEE1394 interface board (I-O Data Device, Ishikawa,Japan).
As for software, the PC was equipped with theWindows 7 Professional operating system (OS), while

Published by Atlantis Press
Copyright: the authors
149

Emotion Recognition of a

Visual C++ 6.0, and Visual C++ 2008 Express Edition,(Microsoft, Redmond, WA) were used as theprogramming languages.
To generate a thermal image that would allow theface area of a subject to be easily extracted, 256 graylevels were set to cover the detectable temperature rangeof our experimental device setup. The visual and audio
information was saved in the PC as a Type 2 digital
video-audio video interleave (DV-AVI) file, in which
the video frame had a spatial resolution of 720 × 480
pixels and 8-bit gray levels, while 48 kHz and 16-bit
level sound was saved in a stereo pulse-code modulation(PCM) format.
Subject A, a male wearing eyeglasses, deliberately
performed each of the emotions of “anger,”“happiness,” “neutrality,” “sadness,” and “surprise,”while speaking the semantically neutral utterance ofeach of the Japanese first names listed in Table 1.
However, all utterances of the Japanese first name“taro” (the first and last vowels of which are /a/ and /o/)and “tsubasa” (the first and last vowels of which are /u/and /a/) were excluded because they had been used in
the previous study.5
In this study’s experiment, Subject A intentionally
maintained a front view in the AVI files, which weresaved as both training and test data. In total, 15 training
data and 15 test data samples were assembled. The AVI
files were used for measuring the facial expression
intensity. WAVform Audio format (WAV) files
obtained from the AVI files were used for measuring
the utterance times.

76.5%, which was near 80% for “taro” and “tsubasa”
used in the previous experiment.5
Table 3 shows the emotion recognition accuracy
values for each utterance obtained using our method.
The emotion recognition accuracy for each utterance
was from 60% to 100%. In the case of “sadness,” theaccuracy was 100%, primarily because the utterance time
was remarkably longer than for other utterances, which
made it easy to distinguish. In contrast, the emotions of“anger,” and “surprise,” were confused with each othermuch more often, primarily because the utterance times
when expressing these two emotions were quite similar.
Table 2. Emotion recognition accuracy.
Input emotion
Angry Happy Neutral Sad Surprised
Angry 86.96
Happy
Recognized
Neutralemotion
Sad
Surprised 13.04

Last
vowel

aiu
eoa

ayaka
kazukitakerukaede-ishinnyahikarishigerumisae
hirokoyuki
fuyu
yusukeyuto

keita
megumimegu
keisukekeiko

3.2. Results and discussion
Our results show that the subject’s emotion wasreflected in his thermal face image even at 0.3 s before
he began to speak.5 From a comparison with the data
analyzed at the time of utterance, we found that the timeseries facial expression intensity differences for the fivetargeted emotions became more distinct when theanalysis included the time range beginning 0.3 s before
to the subject started to speak to 0.3 s after his finishingspeaking.5
Table 2 shows the emotion recognition accuracy
values obtained using our method. As can be seen inthis table, the average emotion recognition accuracy was

100
56.51

4.35

Table 3. Emotion recognition accuracy for each utterance.
NSaSuAccuracy(%)

Su HSuSaSu60

kazukiASaNSaSu80

takeruSu NNSa

Su

60

ayaka

Recognizedemotion foreach utterance

Input emotion
HUtterance

otomoyakojinoboru
kozue
tomoko

26.09
8.7
8.7

13.04
82.61

(%)

Table 1. Japanese first names used in the experiment.
First vowel
ue

4.35
56.51
26.09
8.7
4.35

Akaede

A

H

N

Sa

Su

100

shinnya

A

N

N

Sa

A

60

hikari

ASa

N

Sa

Su

80

shigeru

AH

N

Sa

A

80

misae

A

N

N

Sa

Su

80

hiroko

AH

NSa

Su

100yuki

A

H

N

Sa

A

80

fuyu

Su H

NSaA

60

yusuke

AHH

Sa

Su

80

yuto

A SuNSaA

60keitaA

NN

Sa

N60megumi

A

HHSa

A

60meguA

HNSaSu100keisuke

AHHSaSu80

keikoANNSaSu80

tomoya

ANN

Sa

Su

80

kojiA

A

NSaN60noboru

AHN

Sa

H80

kozue

A

H

N

Sa

H

80

tomoko
A HN
Sa Su
100
A=Angry, H=Happy, N=Neutral, Sa=Sad, Su=Surprised

Published by Atlantis Press
Copyright: the authors
150

Yuuki Oka, Yasunari Yoshitomi, Taro Asada, Masayoshi Tabuse

Standardized utterance timeFig. 3 shows two-dimensional center-of-gravitydistributions for the dominant cluster of each emotion
class for both the training and test data samples. Whenthe nearest neighbor rule is used, it can be seen thatemotion recognition accuracy was 100% and 60% for
the utterance of “kaede” and “fuyu,” respectively.
angry
happy
neutralsadsurprisedtraining
◇○△
□
×test
◆
●
▲
■
＊

Standardized utterance timeStandardized facial expression intensity

Standardized facial expression intensity

Fig. 3. Two-dimensional center-of-gravity distribution of majorcluster of training and test data; upper: “kaede”, lower: “fuyu”.

4. Conclusion

Acknowledgements
The present study wasKAKENHI (22300077).

partiallysupportedbyReferences
1. Y. Yoshitomi, N. Miyawaki, S. Tomita, and S. Kimura,
Facial expression recognition using thermal image
processing and neural network, in Proc. 6th IEEE Int.
Workshop on Robot and Human Communication, (Japan,Sendai, 1997), pp. 380–385.
2. Y. Yoshitomi, T. Asada, K. Shimada, and M. Tabuse,
Facial expression recognition of a speaker using vowel
judgment and thermal image processing, J. Artif. Life andRobotics 16(3) (2011) 318–323.
3. Y. Yoshitomi, M. Tabuse, and T. Asada, Facialexpression recognition using thermal image processing,in Image processing: methods, applications andchallenges ed. V. H. Carvalho (Nova Science Publisher,New York, 2012), pp. 57–85.
4. Y. Nakanishi, Y. Yoshitomi, T. Asada, and M.Tabuse,
Facial expression recognition using thermal image
processing and efficient preparation of training-data, J.
Robotics, Networking and Artif. Life 2(2) (2015) 79-84.
5. Y. Yoshitomi, T. Asada, R. Kato, and M. Tabuse, Facial
expression recognition using facial expression intensity
characteristics of thermal image, J. Robotics, Networkingand Artif. Life 2(1) (2015) 5-8.
6. Y. Yoshitomi, T. Asada, R. Kato, and M. Tabuse,Method of facial expression analysis using video phone
and thermal image, J. Robotics, Networking and Artif.
Life 1(1) (2014) 7-11.
7. T. Asada, Y. Yoshitomi, A. Tsuji, R. Kato, M. Tabuse, N.
Kuwahara, and J. Narumoto, Facial expression analysis
while using video phone, J. Robotics, Networking andArtif. Life 2(4) (2016) 258-262.

In a previous study, we proposed a method forrecognizing human emotions. Using that method, theemotions of one subject were discernable with 76.5%
accuracy in speaking each of 23 utterance types while
intentionally exhibiting each of the five emotions of“anger,” “happiness,” “neutrality,” “sadness,” and“surprise.”
The recorded accuracy level was near 80% for“taro” and “tsubasa” in the previous experiment.5 These
results show that the method is capable of producing
good results.

Published by Atlantis Press
Copyright: the authors
151