SUGISAKAMASANORI

Journal of Advances in Artificial Life Robotics
Vol. 1(2); September (2020), pp. 60–64
ON LINE ISSN 2435-8061; ISSN-L 2435-8061
https://alife-robotics.org/jallr.html

Research Article

Facial Expression Synthesis System Using Speech Synthesis
and Vowel Recognition
Taro Asada1, Ruka Adachi2, Syuhei Takada3, Yasunari Yoshitomi1, Masayoshi Tabuse1

Graduate School of Life and Environmental Sciences, Kyoto Prefectural University, 1-5 Nakaragi-cho, Shimogamo, Sakyo-ku, Kyoto 606-8522, Japan
Second System Department, Software Service, Inc., 2-6-1 Nishi-Miyahara, Yodogawa-Ku, Osaka, Japan
3Resident Department, Seika Town Hall, 70 Kitashiri, Minamiinayazuma, Nishi-Miyahara, Sagara-Gun, Kyoto, Japan
1
2ARTICLE INFO
Article History
Received 14 November 2019
Accepted 20 July 2020
Keywords
MMD Agent
Speech recognition
Vowel recognition

ABSTRACT

Herein, we report on the development of a system for agent facial expression generation that uses
vowel recognition when generating synthesized speech. The speech is recognized using the Julius
high-performance, two-pass large vocabulary continuous speech recognition decoder softwaresystem, after which the agent’s facial expression is synthesized using preset parameters that
depend on each vowel. The agent was created using MikuMikuDanceAgent (MMDAgent), which
is a freeware animation program that allows users to create and animate movies with agents.

© 2022 The Author. Published by Sugisaka Masanori at ALife Robotics Corporation Ltd.
This is an open access article distributed under the CC BY-NC 4.0 license(http://creativecommons.org/licenses/by-nc/4.0/).

Speech synthesis

In this study, to make the agent's performance on apersonal computer (PC) screen more human-like, wehave developed a system for agent facial expression
generation that uses vowel recognition when generating
synthesized speech.

1. Introduction
In Japan, society is aging at an unprecedentedly rapid
pace, the average age of the population has been
increasing, and these trends are expected to continue.
Because of this, researchers have been studying ways ofapplying information technology (IT) to improving themedical and/or mental support provided to older adults,
including persons with extreme psychiatric disorders.
In our previous study [1], we developed a system foranalyzing the facial expressions of a person obtained
while answering interview questions posed by ananimated agent. To accomplish this, we used
MikuMikuDanceAgent (MMDAgent) [2], which is afreeware animation program that allows users to create
and animate movies with agents. To produce the agent’svoice, we used the speech synthesis function setting built
into MMDAgent. Agent motions during the interview
included "nodding" and "blinking" movements.

2. Proposed System and Method
2.1. System overview and outline of the method
Figure 1 shows the processing flow of this system, whichconsists of six processing units:• creating facial expression data,• recording voice utterances,• automatic WAVE file division,• speech recognition by the Julius high-performance,
two-pass large vocabulary continuous speech
recognition decoder software [3],• insertion of expressionless data,• creating facial expression motion.

Corresponding author’s E-mail: t_asada@mei.kpu.ac.jp, yoshitomi@kpu.ac.jp, tabuse@kpu.ac.jp
URL:http://www2.kpu.ac.jp/ningen/infsys/English.index.html

60

3.2. Agent voice recording

Creating facial expression data↓
Recording agent voice↓Automatic division of WAVE files↓
Speech recognition using Julius↓
Inserting data of agent neutral facial expression↓Creating motion data for facial expression

In our system, utterance contents are input as text andused by the MMDAgent to output synthesized voice thatis then recorded by a stereo mixer inside a PC and saved
as a WAVE file.
3.3. Automatic division of WAVE files
After all the utterances have been recorded, if there aremultiple questions, the WAVE file is automaticallydivided for each question, and a new WAVE file iscreated for each question.

Fig. 1. Processing flow of system.

The facial expression data are created in advance. Other
five processing units are successively performed in theorder described above.
3. Development of facial expression synthesis
system
3.1. Creating facial expression data
Expression motions are generated by combining theexpression data of each vowel for each utterance motion.
Facialexpression
data
werecreatedwith
MikuMikuDance [4]. In this study, in order to realize
more human-like agent facial expressions, facial
expression data are created for the vowels / a /, / i /, / u /,/ e /, and / o /, which correspond to all vowels existing inJapanese. Figure 2 shows facial expression of the agent
in uttering each vowel.

/a//i/Fig. 3. Results of speech recognition by Julius.

/u//o//e/Fig. 2. Facial expression of the agent in uttering each vowel.

Fig. 4. An output file listing vowels, and their start and endtimes.

61

3.4. Speech recognition using Julius

Reading audio data from Julius out file↓Calculating the total number of bones andwriting it to the file header.
↓Reading facial expression dataaccording to vowel uttering time↓
Change the vowel uttering time to the number of frames andwriting corresponding data to the vmd file

The results of speech recognition using Julius are shownin Figure 3. From the recognition results, which also
includes the information of consonants, an output file iscreated that only lists the vowels, and their start and end
times (Figure 4).
3.5. Inserting neutral agent motion data

Fig. 6. Flow of facial expression motion creation.

In order to create more natural agent facial expressions,processing is then performed to insert a neutral facial
expression when the vowel / a / is continuous. Figure 5
shows the insertion of neutral motion data executed when
“いかがですか?” (“ikaga desu ka?”), which means,“How is it ?”, is spoken in Japanese. Since the vowel / a/ is continuous when “ka” and “ga” are uttered, neutral
data are inserted between them.

4. Experiment4.1. Conditions
The experiment was performed on a Dell Inspiron 15 PC
equipped with an Intel Core i7-6700HQ 2.2 GHz central
processing unit (CPU) and 8.0 GB of random accessmemory (RAM). The Microsoft Windows 7 Professional
operating system (OS) was installed on the PC, andMicrosoft Visual C++ 2010 Express was used as thedevelopment language.
An animated agent that utters eight questions usedin the initial diagnosis of depression by psychiatrists(Table 1) was created under two conditions (Condition 1:
Created manually, Condition 2: Created with this system).
In addition, an animated agent that asked nine questions(Table 2) on the Hasegawa Dementia Scale [5], which isused in the diagnosis of dementia, was also created using
our system (Condition 3).
The content of the questionnaire is “How was theagent's mouth movements?” and the answer options werefrom a five-point scale (5: very natural, 4: natural, 3:normal, 2: unnatural, 1: very unnatural).

3.6. Creating motion data for a facial expression
Figure 6 shows the flow of creating a facial expression
motion. The facial expression motion data, in the form ofa Vocaloid Motion Data (vmd) file, is created bycomposing the vowel vmd file based on the speech
duration (Sect. 3.4, Figure 4). First, the number of bonesof the standard wire frame model to be used is calculated
from the number of vowels and neutral frames, and thetotal number is written in the vmd file header. Next, thevowel voicing time is converted to the number of frames(1 frame = 1/30 seconds) in order to set each bone of the
facial expression data and is then written to the vmd file.

Table 1. Depression diagnosis test.
No.
/a/1/a/2
34
5
6
78Fig. 5. Insert neutral data while vowel / a / is continuously uttered.

62

Question
Do you get depressed or feel gloomy in a dailylife?
Do you feel less motivated or lacking energy?
Do you get enough sleep at night?
How is your appetite?
Do you enjoy any hobbies on your day off?
Do you feel worthless for yourself or hopeless on
your alive days?
How about concentration and attention to work?
Do you work efficiently?

Table 2. Dementia diagnosis test.
No.
1
2
3
4
5
6
7
8
9Question
How old are you?
What day of the month, what day, what day is it?
Where are we now?
Please tell me the three words you will say.
Please remember it later.
Please subtract by seven in continuous orderusing 100 as the starting value.
Please tell me the numbers I will say from thereverse. 6-8-2.
Please say the words that you learned earlier.
I will show you five items. Tell me what
happened because I'll hide it.
Please say as many vegetable names as youknow.

8
2.57
3.00

44.00

53.57

Condition
36
3.86

7
3.71

8
3.86

94.00

Ave.
3.78

The authors would like to thank Professor J. Narumoto ofthe Kyoto Prefectural University of Medicine for his
valuable support and helpful advice during the course ofthis research. We would also like to thank the subjects ofour experiments for their cooperation. This research wassupported by the Center of Innovation Science andTechnologybased
Radicalinnovation
andEntrepreneurship Program (COI STREAM) of the
Ministry of Education, Culture, Sports, Science andTechnology of Japan.

Condition
12
3
4
1
2.29
2.71
2.86
3.57
2
3.29
3.71
3.29
3.86
7
2.43
2.57

3
3.57

Acknowledgments

Table 3. Average impression evaluation value ofall subjects under Conditions 1 and 2.

6
3.43
3.57

23.86

Herein, we reported on the development of a system foragent facial expression synthesis generation that usesvowel recognition when generating synthesized speech.
The speech is recognized using the Julius highperformance, two-pass large vocabulary continuous
speech recognition decoder software system, after whichagent facial expression synthesis is performed usingpreset parameters depending on each vowel sound.
To create the agent, we used MMDAgent, which isa freeware animation program that allows users to create
and animate movies with agents. To produce the agent’svoice, we used the speech synthesis function setting built
into MMDAgent. The impression evaluation values
obtained from a questionnaire survey indicate that anagent produced by our proposed system is more naturalthan an agent created using preset parameters manually
decided for each utterance. In the future, we plan to use
this system for facial expression analysis and speech
analysis experiments. It is believed that this system could
prove more useful in the treatment of persons sufferingfrom psychiatric ailments such as depressive disorder anddementia than our reported system.1

Table 3 shows the average impression evaluation valuesof all subjects under Conditions 1 and 2. In Table 3,headings “1” to “8” indicate items by question no., andthe “average value” is obtained by averaging over allquestions. In the average impression evaluation values
shown in Table 3, Condition 2 was higher than Condition
1 for all questions, and the “average value” was about
18% higher. From these results, we confirmed that oursystem (Condition 2) has an advantage over manual work
(Condition 1).
Table 4 shows the average impression evaluation
values of all subjects under Condition 3. In Table 4,headings “1” to “9” indicate items by the question no.,and the “average value” is obtained by averaging over allquestions. From this table, it can be seen that because theanimated utterances of the dementia diagnosis questions
also obtained high impression evaluation values, theversatility of our system has been demonstrated.

5
2.71
3.29

1
3.57

5. Conclusion

4.2. Results and discussion

Condition
12

Condition
3

Ave.
2.82
3.32

References
1. T. Asada, D. Kogi, R. Shimada, Y. Yoshitomi, and M.
Tabuse, A System for Analyzing Facial Expression andVerbal Response of a Person while Answering Interview
Questions by Agent, in Proc. of Int. Conf. on Artif. Life andRobotics (Beppu, Oita, Japan, 2018), 534-537.
2. MMDAgent, http://www.mmdagent.jp/ Accessed 6 July 2019.

Table 4. Average impression evaluation value ofall subjects under Condition 3.

63

3. Julius, http://Julius.osdn.jp/, Accessed 6 July 2019.
4. MikuMikuDance, https://sites.google.com/view/vpvp/,
Accessed 9 December 2019.
5. Chiba medical association Revised Hasegawa's dementia
scale(HDS-R)
https://www.jpn-geriatsoc.or.jp/tool/pdf/tool_05.pdf, Accessed 6 July 2019.

Dr. Yasunari YoshitomiHe received his B.E, M.E.
and Ph.D. degrees from
Kyoto University in 1980,
1982 and 1991, respectively.
He works as a Professor at
the Graduate School of Lifeand Environmental Sciencesof Kyoto Prefectural University.
His specialties are appliedmathematics and physics,informatics
environment,intelligent informatics. IEEE, HIS, ORSJ, IPSJ, IEICE,SSJ, JMTA and IIEEJ member.

============================================

Authors Introduction

Dr. Taro AsadaHe received his B.S., M.S.
and Ph.D. degrees from
Kyoto Prefectural Universityin 2002, 2004 and 2010,
respectively. He works as anAssociate Professor at theGraduate School of Life andEnvironmental Sciences ofKyoto Prefectural University.
His current research interests
are human interface and image processing. HIS, IIEEJ
member

Ms. Ruka Adachi

Dr. Masayoshi Tabuse

He received his M.S. andPh.D. degrees from Kobe
University in 1985 and 1988
respectively. From June
1992 to March 2003, he had
worked in Miyazaki University.
Since April 2003, he has
been in Kyoto PrefecturalUniversity. He works as a
Professor at the Graduate
SchoolofLifeandEnvironmental Sciences of Kyoto Prefectural University. His
current research interests are machine learning, computer
vision and natural language processing. IPSJ, IEICE andRSJ member.

She received her B.S. degreefrom Kyoto Prefectural
University in 2018. She
works at Software Service,Inc.

============================================
Mr. Syuhei Takada

He received his B.S. degreefrom Kyoto Prefectural
University in 2019. Heworks at Seika Town Hall.

64