3-5-2

スペシャル・セッション〔国際セッション〕

Objective evaluation of the DNN-based dialog speech synthesizer withdimensional control of emotion ∗☆ Masaki Yokoyama, Tomohiro Nagata and Hiroki Mori (Utsunomiya University)

1Introduction

thetic speech by objective evaluation.

To make communication between human and machine closer to one between humans, speech synthe-

2Natural dialog speech corpus
The UU Database [6] is a speech corpus for study-

sis should be able to express emotions and attitudes

ing linguistic and phonetic phenomena in expres-of speakers as paralinguistic information.
Most studies on speech synthesis considering emo-sive spoken dialog. The database consists of natural

tions are based on the basic emotion theory [1].

dialogs spoken by seven pairs of college students.

However, human emotions are not so simple as to be

The task of the dialogs is “four-frame cartoon sort-

explained by basic emotions alone. One of the de-ing.” Thanks to the amusing nature of the task,

scription methods of emotion is the dimensions [2].

the database is characterized by a wide variety ofWith this method emotion can be described in morerecorded expressive dialog speech.
A major feature of the UU Database is thatdetail than with emotion categories.
Most, speech corpora used in speech synthesis re-paralinguistic information represented by a six-search is read-style [3]. However, speech in con-dimension vector is given for each utterance. The

versation is also diﬀerent from read speech in that

dimensions are pleasantness, arousal, dominance,

it conveys speaker’s emotion and attitude.

For

credibillity, interest and positivity. Paralinguistic

this reason, we believe that read speech corpora isinformation was annotated by three qualified anno-

inadequate for reproducing human communication

tators on a 7-point scale for each dimension. Forby speech synthesis, which leads us to investigate

example, for the dimension of pleasantness, 1: ex-speech synthesis using natural dialog speech corpus.

tremely unpleasant, 2: very unpleasant, 3: somewhat

Previously, we studied a dialog speech synthesis

unpleasant, 4: neutral, 5: somewhat pleasant, 6: very

based on multiple-regression hidden semi-Markov

pleasant and 7: extremely pleasant.

model (MRHSMM) [4]. Although the MRHSMM
enabled to control paralinguistic information in theform of dimensions such as pleasant-unpleasant,aroused-sleepy, etc., synthesized speech tended tohave extreme parameters due to badly estimated regression matrices. We have shown that MAP estimation of regression matrices was eﬀective to reduce
the overfitting problem [4]. However, the problem
still remains for certain combinations of given inputof paralinguistic information.
In recent years, on the other hand, neural networks (NN) took place of HMMs for modelingcontext-dependent acoustic parameters for speechsynthesis [5]. Incorporating neural network is expected to improve the quality of synthetic speech,
as well as the controllability of paralinguistic information.
In this paper, we propose a method of controlling
paralinguistic information in neural network-based
dialog speech synthesis. And then, we studied syn∗3

Paralinguistic information controlin DNN speech synthesis
In speech synthesis, the role of neural network is

to model the relationship between linguistic features
and acoustic parameters of speech. In this paper, wealso model the dependency of acoustic parameterson paralinguistic information, as well as linguisticfeatures.
This is achieved by giving paralinguistic information in the form of dimensions into the input layer,
as shown in Fig. 1. For example, giving low pleasantness and high arousal to the input layer as paralinguistic information would change the output acoustic parameters to those of angry or irritated utterances. At the time of training, averaged value overthree annotators, provided by the UU Database, wasfed to an input unit of each dimension. At the timeof synthesis, giving arbitrary paralinguistic informa-

感情の次元制御による DNN 対話音声合成の客観評価 , 横山雅季, 永田智洋, 森大毅 (宇都宮大)

日本音響学会講演論文集- 1271 -2019年3月

100・・・

Fundamental frequency [st]Linguistic Paralinguisticfeaturesfeatures
・・・

・・・

・・・

・・・・・・・・・

Acoustic parameters
Fig. 1: Network architecture.
tion to the input layer will make the synthesized
speech reflect the specified paralinguistic informa-

9590
Method85

conventional(MRHSMM)
proposed(DNN)3̲3 4̲3 5̲3 3̲4 4̲4 5̲4 3̲5 4̲5 5̲5Given paralinguistic information(Plesantness_Arousal)
Fig. 2: Changes in the mean value of the fundamental frequency with changes in paralinguistic information.
learning rate and batch size of the phoneme dura-tion.

tion model and the acoustic model was set to 0.001

4Paralinguistic information control

4.1

Model structuresand 256. The dropout was set to 0.20 for the input
layer and 0.50 for the others.

Model training and synthesis was performed us-5

Objective evaluation

ing the speech synthesis tool nnmnkwii [7] and Pylayer are represented by a binary vector express-The eﬀect of manipulating input paralinguistic information on the output acoustic parameters withing the type of phoneme, the accent position, thethe proposed deep neural network (DNN) was in-number of morae, and so on, which has 385 dimen-vestigated. For the test, 94 utterances that do notsions. In addition to these, a 4-dimension vector

overlap the training set were synthesized with thethat represents the relative position of phonemes inconventional MAP-MRHSMM and proposed DNN.

frames is given to the input of the acoustic model. InFig. 2 shows the change in the distribution of av-

this study, two additional dimensions are appended
as paralinguistic information. Therefore, the input

eraged fundamental frequency of synthesized utter-layer of the duration model and the acoustic model

guistic information. Each pair of values along thehas 387 and 391 dimensions, respectively.

horizontal axis indicates the given pleasantness andTorch [8]. The linguistic features given to the input

ances, with given 2-dimensional values of paralin-

The output of the acoustic model consists ofarousal to the input (4: neutral). The result for112 dimensions, which include logarithmic funda-

MRHSMM shows that the fundamental frequency

mental frequency, band averaged aperiodicity, 35th-gets higher as the input to the arousal dimension

order mel-cepstrum coeﬃcients, voiced/unvoiced,

is changed to more aroused (e.g.

and their corresponding dynamic features.

4 5). Considering that the positive correlation be-

43 → 44 →The duration model and the acoustic model have atween arousal and F0 was repeatedly mentioned incommon structure with four fully-connected hidden

the literature [10, 11], the tendency shown in Fig. 2

layers, each of which has 2048 units. In both mod-is reasonable. The result for DNN also shows sim-els, dropout was applied before the output layer toilar tendency. However, the correlation for DNN isprevent over fitting.

not as clear as that for MRHSMM.
In both methods, changing the pleasantness di-4.2

Model training

mension (e.g. 3 4 → 4 4 → 5 4) does not show ap-

For the model training, 559 utterances of one fe-parent eﬀects on F0. A two way ANOVA (pleasant-male speaker were used. The sampling frequency
was 16 kHz. World [9] was used for extraction ofness(3) × arousal(3)) for DNN revealed a significant

acoustic parameters.

In the training, 1% of theutterance was held out for model evaluation. The

日本音響学会講演論文集

main eﬀect of pleasantness (F (2, 845) = 138.567,
p < 0.01) and arousal (F (2, 845) = 1066.680, p <0.01).

- 1272 -2019年3月

Spectral tilt [dB/3000Hz]30

p < 0.01). A simple test revealed that pause duration was influenced by pleasantness, only aroused
was low level (3 and 4).
From these results, it can be concluded that the20

paralinguistic information given to the input layer is10reflected in the acoustic parameters of synthesized
speech.

Method0conventional(MRHSMM)
proposed(DNN)3̲3 4̲3 5̲3 3̲4 4̲4 5̲4 3̲5 4̲5 5̲5Given paralinguistic information(Plesantness_Arousal)

Fig. 3: Changes in spectral tilt at vowel /a/ with

6

Subjective Evaluation

6.1

Experimental Conditions

To confirm the eﬀectiveness of the proposed paralinguistic information control method for dialog

changes in paralinguistic information.

speech synthesis, a subjective evaluation test wasconducted. The eﬀectiveness was evaluated from theFig. 4: F0 contours for diﬀerent combinations ofthesized with MRHSMM and DNN (proposed). The

paralinguistic information.

stimuli also include the utterances synthesized withF0 [st]105 /u/ /N/ /t/ /o/ /n/ /e/ /s/ /o/ /n//o/
100
95
90
5_5
4_4
85
3_3
80
20 40 60 80 100 120 140 160
0Number of Frames

Changes in spectral tilt of vowels /a/ are shownin Fig. 3.

The result shows that the spectral

tilt gets shallower as the input to the pleasantness dimension is changed to more pleasant. Because steep spectral tilt makes speech softer, it canbe said that unpleasantness makes the synthesized
speech soft. A two way ANOVA (pleasantness(3)× arousal(3)) for DNN revealed a significant maineﬀect of pleasantness (F (2, 845) = 52.785, p < 0.01)
and arousal (F (2, 845) = 7.308, p < 0.01).
An example of DNN-synthesized F0 contours for atest utterance with diﬀerent paralinguistic information is shown in Fig. 4. The sentence was ‘uNtone
sono’ (“um the ...” in Japanese). Comparing theF0 contours with diﬀerent paralinguistic features,
higher-pitched and longer utterance is synthesized
for more pleasant and more aroused input.
Although there is no correlation between pleasantness/arousal and phoneme duration, there does exist

following two aspects:
1. naturalness
2. controllability of paralinguistic information
The stimulus set was composed of utterances syn-a conventional DNN with the same structure as
the proposed DNN except that it does not accept
the paralinguistic information as input. Ten utterances were selected from the test set for the evaluation. The set of target paralinguistic informationwas the nine combinations of pleasantness (3, 4, 5)
and arousal (3, 4, 5). Hence, the number of stimuli
was 3 (methods) × 10 (test sentences) × 9 (control vectors) = 270. 13 subjects participated in theexperiments. Stimuli were presented to each subject through headphones in a quiet laboratory. Eachstimulus was presented only once to the subjects.
The subjects were asked first to evaluate the naturalness of each stimulus on a 5-point scale, then to
the evaluate perceived paralinguistic information foreach stimulus on a 7-point scale, in the same way asevaluating natural utterances in the UU Database
[6].
6.2

Experimental Results and Discussion

a positive correlation between pleasantness/arousal

The distribution of mean opinion score (MOS)

and pause duration. A two way ANOVA (pleas-for the naturalness is shown in Fig. 5. The aver-antness(3) × arousal(3)) for DNN revealed a signifi-

age of naturalness for MRHSMM, DNN (conven-cant main eﬀect of pleasantness (F (2, 845) = 63.383,tional), and DNN (proposed) was 1.98, 3.07 andp < 0.01) and arousal (F (2, 845) = 61.462, p <2.77, respectively. Some utterances synthesized with0.01). Also, a significant interaction between pleas-

MRHSMM were perceived as extremery unnatural,antness and arousal was found (F (4, 845) = 59.602,

typically for some combinations of pleasantness and日本音響学会講演論文集- 1273 -2019年3月

654
32

12

34

567

Given Value(a) MRHSMM (conventional)

2

R = -0.003

Perceived Value

651

2

3

1MOS

34

4

Perceived ValueMRHSMM
DNN (conventional)
DNN (proposed)

7

R = 0.047

75123456

Given Value(b) DNN (proposed)

7

Relationship between ‘Pleasantness’
perceived value and given value
765432123456

7

1

1ment.

Given Value(c) MRHSMM (conventional)

On the other

R = 0.625

Perceived Value

432Fig. 5: Results of naturalness evaluation experi-arousal (3 3, 4 3, 5 3, 3 4, 3 5).

75

Given paralinguistic information(Pleasantness̲Arousal)

R = 0.633

63̲3 4̲3 5̲3 3̲4 4̲4 5̲4 3̲5 4̲5 5̲5Perceived Value

112345

6Given Value(d) DNN (proposed)

7

Relationship between ‘Arousal’perceived value and given value

hand, utterances synthesized with DNN (conventional) and DNN (proposed) were perceived as rela-Fig. 6: Perceptual experiment results of paralinguis-

tively natural, regardless of given paralinguistic in-tic information control.

formation.
It can be seen that the utterances synthesizedtion. On the other hand, the DNN-based methodwith DNN (conventional) has higher naturalness

provides the controllability of paralinguistic infor-

than those synthesized with DNN (proposed). Fur-

mation in the form of emotion dimensions, without

ther investigation revealed that the diﬀerence of nat-sacrificing the naturalness.

uralness mainly came from the duration. We do notthink the diﬀerence is essential, because this phenomenon disappears with a careful treatment of network initialization.

References[1] P. Ekman, Nebraska Symposium on Motivation, 19, 207–282, 1972.

Fig. 6 shows the distribution between given andperceived values of paralinguistic information, which[2] J. Russell, J. Pers. Soc. Psychol., 39, 1161–1178, 1980.

corresponds to the controllability of paralinguistic
information. Pleasantness seems hardly controllable[3] Yamagishi et al., IEEE Transactions on Audio,with both methods. On the other hand, regard-Speech and Language Processing, 17, 1208–1230.

ing the control of arousal, the correlation coeﬃcient with the proposed DNN was almost the sameas that of the conventional method. For reference,
R = −0.075 for the conventional DNN that does not
accept paralinguistic information as input.

[4] Nagata et al., Speech Commun., 88, 137–148,
2017.
[5] Zen et al., Proc. ICASSP 2013, 7962–7966,

From these results, it can be concluded that the2013.

arousal perceived from synthesized speech can becontrolled with the proposed DNN, which at thesame time provides the improvement in naturalness.

7[6] Mori et al., Speech Commun., 53, 36–50, 2011.
[7] https://github.com/r9y9/nnmnkwii[8] https://github.com/pytorch/pytorch

Conclusions
In this paper, we compared the MRHSMM andDNN as paralinguistic information control methods[9] M. Morise, Speech Commun., 84 ,57–65 ,2016.
[10] K. Scherer, Psychological Bulletin, 99, 143–165, 1986.

for dialog speech synthesis. The analysis of synthesized speech revealed that MRHSMM tends to be[11] Grimm et al., Speech Commun., 49, 787–800,

more sensitive to changes in paralinguistic informa-

日本音響学会講演論文集- 1274 -2007.

2019年3月