The NITech text-to-speech system
for the Blizzard Challenge 2018

Kei Sawada1,2, Takenori Yoshimura1,
Kei Hashimoto1, Keiichiro Oura1,
Yoshihiko Nankaku1, Keiichi Tokuda1
1

Nagoya Institute of Technology (NITech)2
Microsoft Development Co., Ltd.
Blizzard Challenge 2018 Workshop on Sep. 8, 2018

Background¢ Text-to-speech (TTS) systems

TTS systems are used in various applicationsu Demand for TTS systems is increasingul High-quality, various speaking styles, various languages, etc.
uSuccess by introducing deep learningl DNN, LSTM, WaveNet, Deep Voice, Char2Wav, Tacotron, etc.

¢ Evaluations of TTS systems

Comparisons are difficult when the training corpus, task, andlistening test are differentu Blizzard Challenge [Black et al. ’05]
ul In order to better understand and compare research techniques inconstructing corpus-based TTS systems with the same data
¢ NITech TTS system for the Blizzard Challenge
uNITech have been submitting a statistical speech synthesis
system to the Blizzard Challenge since 20052

Blizzard Challenge 2015-2018¢ TaskuConstruct a TTS system from children’s audiobooks that issuitable for reading audiobooks to children¢ DatauChildren’s audiobooks were recorded by one female speakerl 2015 (pilot task year): 2 hours
l 2016: 5 hours
l 2017, 2018: 7 hoursuMismatches between speech data and textl Misreading, onomatopoeia, etc.

uSpeech data includes various speaking stylesl Emotions, characters, singing voices, etc.

"I'm king of the jungle," roared Lion.
"I'm going to eat you all up."
"No!" cried the jungle animals.

Character1
Character2
Descriptive part3

NITech 2015-2018 TTS systems¢ NITech 2015 TTS system

Pruning of training data based on ASR
u Introduction of linguistic features based on quotation marksu¢ NITech 2016 TTS system

Automatic construction of training corpus based on ASR
u Introduction of linguistic features based on syntactic parsingu Introduction of DNN acoustic model considering GV trajectoryu¢ NITech 2017 TTS system

Introduction of linguistic features which can predict andreproduce speaking style from text
u Introduction of MDN acoustic model considering GV trajectoryu¢ NITech 2018 TTS system

Introduce pause insertion modelu Introduce WaveNet vocoderu4

NITech 2018 TTS system

5

NITech 2018 TTS system

6

Pause insertion model¢ Pause insertionu

Pause is used as one of emotional expressions⇒ Introduce pause insertion model to reproduce pause insertionstyle of training corpus¢ Pause estimation of training corpus

Phoneme alignment estimation including short pause at all wordboundaries
u Short pause model (HMM with state skip transition)u Duration of estimated short pause is equal to or greater thanthreshold ⇒ word boundary contains a pauseu¢ Pause insertion model

Bi-directional gated recurrent unit (GRU)u Input: linguistic features of word- and sentence-levelu Output: whether or not a pause is inserted after the word (0 or 1)
u7

NITech 2018 TTS system

8

NITech 2018 TTS system

9

WaveNet-based vocoder¢ Frame-level vocoder
uVocoder introduce degradation in speech quality⇒ Introduce neural vocoder¢ WaveNet vocoder [van den Oord et al.; ’16], [Tamamori et al.; ’17]

Directly modeling and generation speech waveformu Modeling speech waveform as classification problem
u Quantization scheme introduces flat white noise⇒ Introduce noise shaping quantizationu[Yoshimura et al.; ’18]

¢ Mel-cepstrum-based quantization noise shaping

Quantization noise considering human auditory
u Apply mel-cepstrum-based prefilter to speech signals
u[Tokuda et al.; ’94]

Realize high-quality speech waveform generation
10

Experimental conditions (1/3)¢ Conditions of training corpus construction

Provided data

1258 pages

Acoustic features

12 dim. MFCC + Δ + ΔΔ

Acoustic model

3 state left-to-right tri-phone GMM-HMM

Language model

Tri-gramTraining corpus for TTS

924 pages¢ Conditions of pause insertion model

Input features

251 dim. linguistic features

Structure of DNN

Bi-directional gated recurrent unit,
3 hidden layers, 128 units, ReLU

Training algorithm

Adam, dropout rate 20%

11Experimental conditions (2/3)¢ Conditions of phoneme duration model

Sampling rate

32 kHzAcoustic features

64 dim. STRAIGHT mel-cepstrum, log F0,
32 dim. mel-cepstrum aperiodicity measure + Δ + ΔΔ

Number of questions

925 questions

Structure of HMM

5 state left-to-right MSD-HSMM¢ Conditions of acoustic model

Sampling rate

32 kHzAcoustic features

64 dim. STRAIGHT mel-cepstrum, log F0
V/UV info., 32 dim. mel-cepstrum aperiodicity measure

Linguistic features

1685 dim.

Structure of DNN

Single-mixture density network,
3 hidden layers, 8000 units, sigmoid

Training algorithm

SGD, dropout rate 60%,
Trajectory training considering GV
12Experimental conditions (3/3)¢ Conditions of WaveNet vocoder

Sampling rate

32 kHzQuantization

8 bit μ-law

Noise shaping parameters

γ =0.1, β=0.1

Structure of WaveNet

Dilation: [1, 2, 4, ..., 512] 3 stacks
Dilation, residual, skip: 256

Condition features of WaveNet98 dim. acoustic features

Training algorithm

Adam

13

DemoThe picture is quoted from the Usborne Publishing. 14Evaluation results of sentence domain
Naturalness
MOSID

Similarity
MOSID

Intelligibility
WER
ID

4.8

A

4.5

A

11I

4.0

K

3.9

K

14E, O

3.7

J

3.6

J

15

D, G

3.5

I

3.5

I

16

K

3.0

L, M

3.4

17

N

2.9

B

3.2

L
B

18J

2.8

D

3.0

M

20

F
...

...

...

A: Natural speech
B, C, D, E: Benchmark TTS systems
I: NITech TTS system
Red line: Significant difference between NITech and other systems
15Evaluation results of page domain
Overall impression
MOS
ID

Pleasantness
MOS
ID

Speech pause
MOSID

StressMOS
ID

48

A

48

A

48

A

48

A

38

K

37

K

K

J, I

33

J, I

K, JI

36

34

36
32

35

J

29

B

28

L, B

31

D, G

33

I

28

L

26

M

30

E

30

D, G
...

...
Emotion
MOS
ID

...

...
Intonation
MOS
ID

Listening effort
MOS
ID

48

A

48

A

49

A

37

K

38

K

37

K

35

J

35

J, I

34

J

33

I

31

B

33

I

28

D30M

28

D...

...

...

16

Comparison of NITech 2017 and 2018
¢ Introduction of WaveNet vocoder

Improved naturalness and speaker similarityu Sometimes ambiguous pronunciation
ul Multiple codecs and noise made WaveNet training difficult
uReduced reproducibility of speaking styles
l Training data is insufficient to reproduce various speaking styles

2017

2018

http://www.sp.nitech.ac.jp/~swdkei/syn/Blizzard_2018/index.html
17

Conclusion¢ NITech TTS system for the Blizzard Challenge 2018

Introduce pause insertion modelu Introduce WaveNet vocoderu Large-scale subjective listening tests
ul Synthesized highly natural, similar, and intelligible speechuComparison of NITech 2017 and 2018 TTS systemsl Improved naturalness and speaker similarity
l Insufficient accuracy of WaveNet vocoder¢ Future work

Generate expressive synthesized speech in neural vocoderu Introduce end-to-end approach
u18