The NITech text-to-speech system
for the Blizzard Challenge 2017

Kei Sawada, Kei Hashimoto,Keiichiro Oura, Keiichi Tokuda
Nagoya Institute of Technology (NITech)
Blizzard Challenge 2017 Workshop on Aug. 25, 2017

Background¢ Text-to-speech (TTS) systems

TTS systems are used in various applicationsu Demand for TTS systems is increasingul High-quality, various speaking styles, various languages, etc.

¢ Evaluations of TTS systems

Comparisons are difficult when the training corpus, task, andlistening test are differentu Blizzard Challenge [Black et al. ’05]
ul In order to better understand and compare research techniques inconstructing corpus-based TTS systems with the same data
¢ NITech TTS system for the Blizzard Challenge
uNITech have been submitting a statistical parametric speechsynthesis (SPSS) system to the Blizzard Challenge since 2005

2

Blizzard Challenge 2017 task
¢ Blizzard Challenge 2017uTask
l Construct a TTS system from children’s audiobooks that is suitablefor reading audiobooks to childrenuData
l 7 hours speech data and text pairsl 56 books were recorded by one female English speakerl Speech data includes various speaking styles, emotions,characters, etc.
l Example of provided data

"I'm king of the jungle," roared Lion.
"I'm going to eat you all up."
"No!" cried the jungle animals.

Character1
Character2
Descriptive part3

Review of Blizzard Challenge 2016
¢ Blizzard Challenge 2016

The task was almost same as the Blizzard Challenge 2017
u Difference was the amount of training data
(2016: 5 hours, 2017: 7 hours)
u5

Mean Opinion Scores (naturalness) − All listeners¢ Result of Blizzard Challenge 2016

MOS for naturalness4ul Training corpus includes various

speaking variations⇒ Modeling is difficult

32Why?

1uScorel MOS of SPSS systems is not so goodn

Redesign of linguistic features
for audiobooks in SPSS

381

381

381

380

381

380

381

380

380

380

381

380

381

381

380

380

380

A

L

M

Q

B

F

H

O

E

DG

JK

PICN

SystemRed: hybrid systems
Blue: unit selection systems
Green: SPSS systems

4

NITech TTS system
Training corpus for SPSS
SpeechText

Vocoder encoding

Text analysis

Acoustic features

Linguistic features

HMM training
Training part
Synthesis partInput text

DNN training
Phoneme-levelalignment

HMM-based
duration model
Text analysis
Linguisticfeature

DNN-based
spectral andexcitation modelParameter generation
Acoustic feature
Vocoder decoding

Synthesized speech¢ Linguistic features for audiobooks in SPSS
¢ Trajectory training considering GV for mixture density networks
5NITech TTS system
Training corpus for SPSS
SpeechText

Vocoder encoding

Text analysis

Acoustic features

Linguistic features

HMM training
Training part
Synthesis part
Input text

DNN training
Phoneme-levelalignment

HMM-based
duration model
Text analysis
Linguisticfeature

DNN-based
spectral andexcitation modelParameter generation
Acoustic feature
Vocoder decoding

Synthesized speech¢ Linguistic features for audiobooks in SPSS
¢ Trajectory training considering GV for mixture density networks
6Linguistic features for audiobooks¢ Linguistic features

Features obtained from texts express pronunciationsu Suitable design for audiobooks is necessaryu¢ Additional linguistic features to HTS-2.3.1 demo scriptuPage-level informationl Capture supra-sentential information

uSyntactic and dependency parsing informationl Capture sentence structureuType of sentencel Distinguish different type of sentenceuDouble quotes informationl Distinguish between descriptive and conversational partsuWord and phrase codesl Distinguish each word and phrase variation

7

Phrase code
¢ Training partA unique value (phrase code) is assigned to each phraseu Phrases are distinguished to represent speaking variationuTraining partTraining corpus
Phrase 1・・・

Phrase 2⁝

Linguisticfeatures

Phrase N
Phrase code (fix-dimensional
normally distributed random vector)

8

Phrase code
¢ Training partA unique value (phrase code) is assigned to each phraseu Phrases are distinguished to represent speaking variationu¢ Synthesis partPhrase is vectorized by using doc2vec [Le et al. ’14]
u Phrase similarity between training and input phrases iscalculated from vectorized onesu Phrase code of the highest similarity phrase is useduTraining corpus
・・・

Linguisticfeatures

The highestsimilarity
phrase code

Synthesis partPhrase 1
Phrase 2⁝Input phrase
doc2vec

Phrase N
Phrase code (fix-dimensional
normally distributed random vector)

9Phrase code
¢ Training partA unique value (phrase code) is assigned to each phraseu Phrases are distinguished to represent speaking variationu¢ Synthesis partPhrase is vectorized by using doc2vec [Le et al. ’14]
u Phrase similarity between training and input phrases iscalculated from vectorized onesu Phrase code of the highest similarity phrase is used
uInput text

Text of phrase adaptation

Synthesizedspeech

Zero vector (average phrase)"I must tellHamlet.”Come and see the friendly lion!
"Who's been sitting in my chair?""I must tell the King.” (highest similarity phrase)

Realize expressive speech synthesis

10NITech TTS system
Training corpus for SPSSSpeech

Text

Vocoder encoding

Text analysis

Acoustic features

Linguistic features

HMM training
Training part
Synthesis part
Input text

DNN training
Phoneme-levelalignment

HMM-based
duration model
Text analysis
Linguisticfeature

DNN-based
spectral andexcitation modelParameter generation
Acoustic feature
Vocoder decoding

Synthesized speech¢ Linguistic features for audiobooks in SPSS
¢ Trajectory training considering GV for mixture density networks
11DNN-based SPSS¢ DNN-based SPSS [Zen et al. ’13]

DNN is trained to represent a mapping function from linguisticfeatures to acoustic featuresu Mixture density network (MDN)-based SPSS [Zen et al. ’14]
ul DNN outputs provide Gaussian mixture model parameters

Inconsistency in training and synthesis criteriau Over-smoothing on speech parameter trajectoriesu¢ Trajectory training considering GV for DNN-based SPSS

[Hashimoto et al. ’16]
Can address inconsistency between training and synthesis
u DNN is optimized considering GVu¢ Trajectory training considering GV for MDN-based SPSS

Expect high-quality acoustic modelu Use a single MDN as the acoustic modelu12

Frame-level training
Frame-levellikelihood

Synthesized speech

N (ot |µt , ⌃t )µtSynthesis partTraining part・・・・・・

ltµ1 µ2 ・・・ µT⌃1 ⌃2 ・・・ ⌃T
Forward-propagation⌃t
Back-propagationc̄1 c̄2 ・・・ c̄T

Forward-propagation

ot

l1

l2・・・

lTv(·) : GV vector
lt : linguistic feature vector ct : static-feature vectorµt : mean vectorc̄t : optimal static-feature vector ⌃GV : GV covariance matrix⌃t : covariance matrix
P : sequence covariance matrix w : GV weight13

Trajectory training considering GV
c1 c2 ・・・ cTv(c) GVlikelihood

N (c|c̄, P ) N (v(c)|v(c̄), ⌃GV )wTc̄1 c̄2 ・・・ c̄TlT

Synthesis part

Forward-propagation
・・・

Training partl2

µ1 µ2 ・・・ µT⌃1 ⌃2 ・・・ ⌃T・・・・・・

Back-propagationc̄1 c̄2 ・・・ c̄Tv(c̄) ⌃GVµ1 µ2 ・・・ µT⌃1 ⌃2 ・・・ ⌃Tl1Synthesized speech

Forward-propagation

Trajectorylikelihood

l1l2・・・lTv(·) : GV vector
lt : linguistic feature vector ct : static-feature vectorµt : mean vectorc̄t : optimal static-feature vector ⌃GV : GV covariance matrix⌃t : covariance matrix
P : sequence covariance matrix w : GV weight
14

TTS system conditions
Training corpus

921 pages

Sampling rage

44.1 kHzFrame

window: F0-adapteve Gaussian, shift: 5 ms

HMM structure

5-state left-to-right MSD-HSMM

Acoustic features(HMM)

49-dim. STRAIGHT mel-cepstrum,
24-dim. mel-cepstrum aperiodicity measure,log F0, and Δ + ΔΔ

Number of questions925 questions

MDN structure

3 hidden layers with 8000 hidden units,
activation function: sigmoid (hidden), linear (output),
dropout rate: 60%, GV weight: 0.001

Acoustic features(MDN)

69-dim. STRAIGHT mel-cepstrum,
34-dim. mel-cepstrum aperiodicity measure,interpolated log F0, voiced/unvoiced information

Linguistic features

925-dim. linguistic features for contexts,
10-dim. duration features,
150-dim. word code, 600-dim. phrase code15

Synthesized speech samples

The picture is quote from the Usborne Publishing.

16

Experimental results¢ Experimental conditions

16 TTS system (+ 1 natural speech)u Results of all participantsu¢ Page domain (60-point MOS)
Criterion

Overall
Pleasant
impression
nessSpeechpause

StressIntonation

Emotion

Listeningeffort

MOS

31

30

31

31

31

33

31

Rank

4th

5th

4rd

4rd

4th

4th

3th

¢ Sentence domain (5-point MOS)
Criterion

Naturalness

Similarity

MOS

3.6

3.0

Rank

3th

7th

¢ Intelligibility test
WER

30%

Rank

1st

Highly natural synthesized speech
17Experimental results¢ Experimental conditions

16 TTS system (+ 1 natural speech)u Results of all participantsu¢ Page domain (60-point MOS)
Criterion

Overall
Pleasant
impression
nessSpeechpause

StressIntonation

Emotion

Listeningeffort

MOS

31

30

31

31

31

33

31

Rank

4th

5th

4rd

4rd

4th

4th

3th

¢ Sentence domain (5-point MOS)
Criterion

Naturalness

Similarity

MOS

3.6

3.0

Rank

3th

7th

¢ Intelligibility test
WER

30%

Rank

1stCompared with MOS of naturalness,
MOS of speaker similarity is low score
18

Experimental results¢ Experimental conditions

16 TTS system (+ 1 natural speech)u Results of all participantsu¢ Page domain (60-point MOS)
Criterion

Overall
Pleasant
impression
nessSpeech
pauseStressIntonation

Emotion

Listeningeffort

MOS

3130

31

31

31

33

31

Rank

4th

5th

4rd

4rd

4th

4th

3th

¢ Sentence domain (5-point MOS)
Criterion

Naturalness

Similarity

MOS

3.6

3.0

Rank

3th

7th

¢ Intelligibility test
WER

30%Rank

1stHighly intelligible synthesized speech
19Conclusion¢ NITech TTS system for the Blizzard Challenge 2017

Linguistic features for audiobooks in SPSSu Trajectory training considering GV for MDN-based SPSSu Large-scale subjective listening tests
ul Synthesized highly natural and intelligible speechl Should improve speaker similarity¢ Future worku

Improve robustness of outliers
l ε-contaminated Gaussian loss [Zen et al. ’16]u

Introduce direct speech waveform prediction models
l WaveNet [van den Oord et al. ’16]

20