Journal of Robotics, Networking and Artificial Life
Vol. 7(2); September (2020), pp. 107–110
DOI: https://doi.org/10.2991/jrnal.k.200528.007; ISSN 2405-9021; eISSN 2352-6386
https://www.atlantis-press.com/journals/jrnal

Research Article

Speech Synthesis of Emotions in a Sentence using
Vowel Features
Rintaro Makino1, Yasunari Yoshitomi2,*, Taro Asada2, Masayoshi Tabuse2
UX Solution Department UX Division, Customer Platform Promotion Division, SoftBank Corp., 1-9-1 Higashi-shimbashi, Minato-ku, Tokyo 105-7317, JapanGraduate School of Life and Environmental Sciences, Kyoto Prefectural University, 1-5 Nakaragi-cho, Shimogamo, Sakyo-ku, Kyoto 606-8522, Japan

12ARTICLE INFOABSTRACT

Article History

We previously proposed a method for adding emotions to synthetic speech using the vowel features of a speaker. For the initialinvestigation in this earlier study, we used utterances of Japanese names to demonstrate the method. In the present study, we use
the proposed method to construct emotional synthetic speech for a sentence formed from the emotional speech of a single malesubject and produce results that are discriminable with good accuracy.

Received 14 November 2019
Accepted 28 April 2020
Keywords
Emotional speechfeature parameteremotional synthetic speechvowelsentence© 2020 The Authors. Published by Atlantis Press SARL.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION

2. PROPOSED METHOD

Recently, methods for adding emotions to synthetic speech havereceived considerable attention in the field of speech synthesis
research [1–8]. To generate emotional synthetic speech, it is necessary to control the prosodic features of the utterances. Natural
language is mainly composed of vowels and consonants. TheJapanese language has five vowels. A vowel has a more dominant
impact on the listener’s impression than does a consonant, primarily because a vowel has a longer utterance time and larger
amplitude in comparison to a consonant. We previously proposeda case-based method for generating emotional synthetic speechby exploiting the characteristics of maximum amplitude andutterance time for vowels, as obtained by using a speech recognition system, together with the fundamental frequency of emotional speech [9].

In the first stage, we obtain audio data for emotional speechrecorded as a WAV file. The subject is asked to speak in a way thatreflects various emotional states: “angry,” “happy,” “neutral,” “sad,”and “surprised.” Then, for each type of emotional speech, we measure the time of each vowel utterance and the value of the maximum amplitude of the waveform while the vowel is being spoken,
as well as the fundamental frequency of the emotional speech [10].

In an earlier study [10], we proposed an approach that furtherimproved the method described in Boku et al. [9] by controllingthe fundamental frequency of the emotional synthetic speech.
The advantage of this earlier study [10] over prior research [1–8]
was the usage of the vowel feature in emotional speech to generate
synthetic emotional speech. In the previous study [10], the speech
included only Japanese names.
In the present study, we apply our previously proposed method[10] to creating emotional synthetic speech in sentence form,
with new treatments for (1) smoothing the fundamental frequencies of sequential vowels and (2) suitably jointing the synthetic phonemes.
Corresponding author. Email: yoshitomi@kpu.ac.jp*In the second stage, we synthesize the phoneme sequence utteredby the subject. This stage consists of the following seven steps:
Step 1: For a vowel with a consonant appearing just before it in thesynthetic speech with neutral emotion, the total phonation duration time of the vowel and the consonant is transformed into thetime for speech with neutral emotion by the human subject. The
synthetic speech obtained by this processing is hereinafter called“neutral synthetic speech” [10].
Step 2: For a vowel with a consonant appearing just before it in thesynthetic speech reflecting one of the intentional emotions (“angry,”“happy,” “sad,” or “surprised”), the total phonation duration time of thevowel and consonant is set so that the ratio of this total to the corresponding total in the neutral synthetic speech is equal to the ratio of
the phonation duration time of the vowel in the emotional speech tothe phonation duration time of the vowel in the neutral speech [10].
Step 3: The fundamental frequency of the synthetic speech obtainedby the processing conducted up through Step 2 is initially adjusted
based on the fundamental frequency of the emotional speech [10].
Step 4: For a vowel with a consonant appearing just before it in thesynthetic speech obtained by the processing conducted up throughR. Makino et al. / Journal of Robotics, Networking and Artificial Life 7(2) 107–110

108

Step 3, the fundamental frequency is transformed into the averageof three values of the sequential vowels that include it. For the firstor last vowel in the sentence, two sets of fundamental frequencies
are used for averaging.
Step 5: Amplitudes are transformed into final values by twice multiplying the ratio (Maxem/Maxne), where Maxem and Maxne denotethe maximum amplitude of the vowel in emotional speech and theamplitude of the vowel in neutral speech, respectively. The synthetic speech obtained by the processing up through Step 5 is hereinafter called “emotional synthetic speech.”
Step 6: The fundamental frequency of the emotional synthetic speechobtained by the processing conducted up through Step 5 is further
adjusted based on the fundamental frequency of the emotional speech.
Step 7: The synthetic speech for a sentence is generated by jointing
all the synthetic speech phonemes constructing the sentence andadjusting the speech length to the phonation duration time of theemotional speech obtained in advance.
If no consonant appears before a vowel, the process described in
Steps 1 through 6 applies only to the vowel.

3. EXPERIMENT
3.1. Condition
We used the speech recognition system known as Julius [11] to recognize the vowels and save the time position of the start and end of
the vowels in each utterance. One male subject (Subject A) in his 50s
was asked to speak the Japanese first names listed in Table 1 in a waythat reflected various emotional states: “angry,” “happy,” “neutral,”“sad,” and “surprised.” The audio data were recorded as WAV files.
In preparation for generating the emotional synthetic speech, wemeasured the utterance time of the vowel, the maximum amplitudeof the waveform, and the fundamental frequency while the vowel
was being spoken. Tables 2–4 show the phonation time average,
the maximum amplitude average, and the fundamental frequency

average, respectively, for each first vowel in each Japanese name ineach emotion category as spoken by Subject A.
Voice Sommelier Neo (premium version; Hitachi Business Solution
Co., Ltd., Yokohama, Japan) [12] was used as the speech synthesizer
for each of the steps described in Section 2. In applying the method,
the Male 1 (bright voice) mode in Voice Sommelier Neo was used.
To convert the amplitude of each vowel and consonant describedin Step 5 in Section 2, a digital audio editor was used. The method[13] using resampling was then used in Step 6 of Section 2.
To enable comparisons, subject A was asked to speak the sentence,‘このぬいぐるみかわいくない’ (in Japanese), which means,‘This stuffed toy is pretty, isn’t it?” in a way that reflected each of theintentional emotions. (“Surprised” was not included here, as it wasdifficult to express in the sentence.) We used the recorded utterances as “emotional speech.” We then generated synthetic speech
for the same sentence (‘このぬいぐるみかわいくない’) using
the method described in Section 2.
In all, 13 subjects participated in the experiment. These included
one male in his 60s, one male in his 50s, one male in his 30s, fivemales in their 20s, and five females in their 20s. Each of the 13 subjects was asked to judge the emotional state of the speaker (angry,happy, neutral, sad) after listening to four utterances, one at a time,in the following order: the emotional speech by Subject A, the emotional synthetic speech. The 13 subjects were also asked which features were very important in judging the emotion of the sentence.
There were seven choices available: three prosodic features, (1)utterance length, (2) height, and (3) volume; and four in-sentence
positional features, (4) top, (5) middle, (6) last, and (7) total. For thethree prosodic features, multiple answers were allowed, while only
one answer was allowed for in-sentence positional features.

3.2. Results and Discussion
Table 5 shows the results of the subjective evaluations of the 13 subjects. As indicated, the mean accuracy of the categorizations for theTable 3 | Maximum amplitude average of each first vowel

Table 1 | Japanese first name used in experiments

EmotionFirst vowel

Last vowel/a//i//u//e//o//a//i//u/

ayaka
kazukitakerukaede
taroshinnyahikarishigerumisae
hirokotsubasayuki
fuyu
yusukeyuto/e//o/

keita
tomoyamegumikoji
megu
noborukeisukekozue
keiko
tomokoTable 2 | Phonation time average of each first vowel

First vowel/a//i//u//e//o/Angry

Happy

NeutralSad

Surprised

1340
362
658
816
748

1714
385
509
1079
1262

714
287
438
794
838

573
199
298
575
646

1346
4001017
1165
1479

Table 4 | Fundamental frequency average of each first vowel

EmotionFirst vowel /a//i//u//e//o/Emotion

Angry

Happy

NeutralSad

Surprised

62
44
48
70
70

110
74
120
126
104

72
68
130
136
118

112
86
86
182
176

58
58
44
84
84

First vowel(ms)/a//i//u//e//o/

Angry

Happy

NeutralSad

Surprised

141
114
159
122
139

183
192
217
186
186

131
134
131
178
170

165
191
165
229
210

284
286
284
276
228
(Hz)

R. Makino et al. / Journal of Robotics, Networking and Artificial Life 7(2) 107–110

emotional speech was 100%; for the emotional synthetic speech,
the mean accuracy was 78.9%. The accuracy for the “happy” synthetic speech was highest, while that of “neutral” was lowest. Table 6
shows the results of the questions regarding the importance ofvarious features in judging the emotional category. As shown,among the three prosodic characteristics, the “height of the voice”
had the most influence, while among the four sentence positions,the “last position” appears to have influenced the subjects the most.
Figure 1 shows the waveforms of the emotional speech and theemotional synthetic speech. As illustrated in the figure, the waveform associated with the “happy” synthetic speech appears to be
the most similar to its emotional speech counterpart among thewaveforms of the four emotional synthetic speech types.
Tables 7 and 8 show the results of the subjective evaluations for thenames “Taro” and “Hiroko,” respectively, as reported in our previous
study [10]. In these two tables, the results for the emotional speech
were calculated as the average of the values obtained in two sets ofTable 7 | Results of the subjective evaluation for “Taro” [10]InputAngry Happy Neutral(1) Emotional speech
97.200094.4
00
2.8
94.4
00
5.6
2.8
2.80

2.802.8
94.4
00
5.60094.4(2) Emotional synthetic speech
Recognition Angry
88.8
5.60Happy
5.6
88.8
0Neutral
0
5.6
94.4
Sad
00
5.6
Surprised
5.6
0000
11.1
88.9
0

11.1
0
00
88.9

Recognition Angry
Happy
NeutralSad
Surprised

(%)

Input

InputHappy

NeutralAngrySad(1) Emotional speech

Happy Neutral Sad Surprised(1) Emotional speech
91.70001000001000008.3
0000010008.3
00091.7

022.2
11.1
66.7
016.6
5.6
0077.8001000000100(2) Emotional synthetic speech
Recognition Angry
76.9
7.7
15.4
Happy
092.3
0
Neutral
23.1
069.2
Sad
0015.4

Recognition Angry
Happy
NeutralSad
Surprised

015.4
7.776.9(2) Emotional synthetic speech
Recognition Angry
83.3
00Happy
066.7
0
Neutral
027.8
94.4
Sad
005.6
Surprised
16.7
5.5
0100000

Recognition Angry
Happy
NeutralSad

010000(%)

Table 6 | Results of questions regarding the importance of various factors
in judging the emotion category
Voice

Sentence position

Length Height Volume
Number of voteRatio (%)

7541185969

Sad Surprised

Table 8 | Results of the subjective evaluation for “Hiroko” [10]Table 5 | Results of subjective evaluations

Angry

109

Top00Middle Last Total
00862538(%)

listening by all 18 subjects. Based on the values in Table 7, for “Taro,”
the mean accuracy for the emotional speech categorizations using
the previously proposed method was 95.0%, while for the emotional
synthetic speech, it was 90.0% [10]. For “Hiroko,” the mean accuracy
for the emotional speech categorizations was 96.7%, while for theemotional synthetic speech, it was 77.8% (Table 8) [10]. Combining
the results for both “Taro” and “Hiroko,” the average accuracy forthe emotional speech categorizations was 95.9%, while the average
accuracy for the emotional synthetic speech was 83.9%.
A comparison of the combined 83.9% average accuracy reported forthe emotional synthetic speech categorizations in the previous study(Tables 7 and 8) [10] and the lower 78.9% average accuracy in the present study [Table 5 (2)] suggests that it may be more difficult to effectively generate emotional synthetic speech in a sentence rather than fora first name (It should be noted that there were four emotion categories
in the more recent experiment versus five in the previous study [10].
As mentioned earlier, “surprised” was omitted in the current study.).

Figure 1 | Waveform of emotional speech and emotional synthetic speech
for the intentional emotions of “angry,” “happy,” “neutral,” and “sad’’ in theutterance of ‘このぬいぐるみかわいくない’ (in Japanese).

4. CONCLUSION
Recently, methods for adding emotion to synthetic speech havereceived considerable attention in speech synthesis research. We

R. Makino et al. / Journal of Robotics, Networking and Artificial Life 7(2) 107–110

110

had previously proposed a method for speech synthesis with emotions using the vowel features of a speaker [10]. In the present study,
we used the proposed method [10] to create emotional synthetic
speech in a sentence based on the emotional speech of an individualand found that the results were discriminable with a good accuracy.

CONFLICTS OF INTEREST
The authors declare they have no conflicts of interest.

ACKNOWLEDGMENTS
We would like to thank all those who participated in the experiments described here.

REFERENCES[1] D. Erickson, Expressive speech: production, perception andapplication to speech synthesis, Acoust. Sci. Technol. 26(2005), 317–325.
[2] A. Iida, S. Iga, F. Higuchi, N. Campbell, M. Yasumura, A
speech synthesis system with emotion for assisting communi­
cation, Trans. Human Interface Soc. 2 (2000), 63–70 (inJapanese).
[3] N. Katae, S. Kimura, An effect of voice quality and control in emotional
speech synthesis, Proceedings of the Autumn Meeting the AcousticalSociety of Japan, vol. 2, Iwate, Japan, 2000, pp. 187–188 (in Japanese).
[4] T. Moriyama, S. Mori, S. Ozawa, A synthesis method of emotional
speech using subspace constraints in prosody, Trans. Inform.
Process. Soc. J. 50 (2009), 1181–1191 (in Japanese).

[5] I.R. Murray, J.L. Arnott, Toward the simulation of emotion insynthetic speech: a review of the literature on human vocal emotion.
J. Acoust. Soc. Am. 93 (1993), 1097–1108.
[6] I.R. Murray, M.D. Edgington, D.M. Campion, J.F. Lynn, Rulebased emotion synthesis using concatenated speech. Proceedingsof ISCA Tutorial and Research Workshop on Speech andEmotion, International Speech Communication Association,
Newcastle, UK, 2000, pp. 173–177.
[7] S. Ogata, T. Yotsukura, S. Morishima, Voice conversion toappend emotional impression by controlling articulation information. IEICE Tech. Rep. Human Inf. Process. 99 (2000), 53–58(in Japanese).
[8] M. Schröder, Emotional speech synthesis—a review, in: P.
Dalsgaard, B. Lindberg, H. Benner, Proceedings of 7th European
Conference on Speech Communication and Technology,
vol. 1, Kommunik Grafiske Losninger A/S, Aalborg, 2001,
pp. 561–564.
[9] K. Boku, T. Asada, Y. Yoshitomi, M. Tabuse, Speech synthesis of emotions using vowel features, in: R. Lee (Eds.), Software
Engineering, Artificial Intelligence, Networking, and Parallel/
Distributed Computing 2012, Springer, Berlin, Heidelberg, 2013,
pp. 129–141.
[10] K. Boku, T. Asada, Y. Yoshitomi, M. Tabuse, Speech synthesis ofemotions using vowel features of a speaker, J. Artif. Life Robot. 19(2014), 27–32.
[11] Julius Development Team, Open-source large vocabulary CSR
engine Julius, Available from: http://julius.sourceforge.jp/(accessed December 11, 2019).
[12] Hitachi Business Solution Co. Ltd., Voice Sommelier Neo,
Available from: https://www.hitachi-solutions-create.co.jp/solution/voice/ (accessed April 28, 2020).
[13] N. Aoki, Sound programming in C, Ohmsha, Tokyo, 2008,
pp. 141–160 (in Japanese).

AUTHORS INTRODUCTION
Mr. Rintaro Makino
He received his B.S. degree from Kyoto
Prefectural University in 2018. He works
at SoftBank Corp.

Dr. Yasunari Yoshitomi
He received his B.E, M.E. and PhD
degrees from Kyoto University in 1980,1982 and 1991, respectively. He works
as a Professor at the Graduate School ofLife and Environmental Sciences of Kyoto
Prefectural University. His specialties areapplied mathematics and physics, informatics environment, intelligent informatics. He is a member ofIEEE, HIS, ORSJ, IPSJ, IEICE, SSJ, JMTA and IIEEJ.

Dr. Taro Asada

Dr. Masayoshi Tabuse

He received his B.S., M.S. and PhD degrees
from Kyoto Prefectural University in 2002,
2004 and 2010, respectively. He works asan Associate Professor at the Graduate
School of Life and Environmental Sciencesof Kyoto Prefectural University. His current research interests are human interface
and image processing. He is a member of HIS, IIEEJ.

He received his M.S. and PhD degrees fromKobe University in 1985 and 1988 respectively. From June 1992 to March 2003, hehad worked in Miyazaki University. Since
April 2003, he has been in Kyoto PrefecturalUniversity. He works as a Professor at theGraduate School of Life and Environmental
Sciences of Kyoto Prefectural University. His current research interests are machine learning, computer vision and natural language
processing. He is a member of IPSJ, IEICE and RSJ.