The NITech text-to-speech system
for the Blizzard Challenge 2016

Kei Sawada, Chiaki Asai, Kei Hashimoto,Keiichiro Oura, and Keiichi Tokuda
Nagoya Institute of Technology (NITECH)
Blizzard Challenge 2016 workshop on Sep. 16, 2016

Background¢ Text-to-speech (TTS) systemsu

TTS systems are used in various applicationsl In-car navigation, smartphones, spoken dialogue systems, etc.

uDemand for TTS systems is increasingl High speech quality, speaking styles, multilingual language, etc.

¢ TTS system based on big data

Quality of synthesized speech is improved by using big datau Speech data recorded with less noise and under same
conditions are suitable for trainingu Recording a large amount of speech data requires a huge costu¢ TTS system based on audiobooks
uAudiobooks can be relatively easily used as a large amount ofspeech data and text pairs2

Blizzard Challenge 2016 task
¢ Blizzard Challenge [Black, et al.; ’05]u

Blizzard Challenge was started in order to better understandand compare research techniques¢ Blizzard Challenge 2016

Task is to construct a TTS system from children’s audiobooksu Five-hours speech data and text pairs are provided
u All 50 books were recorded by one English female speakeru Speech data includes various speaking styles, emotions,characters, etc.
u Example of provided datau"I'm king of the jungle," roared Lion.
"I'm going to eat you all up."
"No!" cried the jungle animals.

Character1
Character2
Descriptive part3

NITech system
Training corpus for SPSS
SpeechText

Vocoder encoding

Text analysis

Acoustic features

Linguistic features

HMM training
Training part
Synthesis part
Input text

DNN trainingPhoneme-levelalignment

HMM-based
duration model
Text analysis
Linguisticfeature

DNN-based
spectral andexcitation modelParameter generation
Acoustic feature
Vocoder decoding

Synthesized speech¢ Automatic construction of training corpus from audiobooks¢ Design of linguistic features for SPSS based on audiobooks¢ DNN-based SPSS

4

NITech system
Training corpus for SPSS
SpeechText

Vocoder encoding

Text analysis

Acoustic features

Linguistic features

HMM training
Training part
Synthesis part
Input text

DNN training
Phoneme-levelalignment

HMM-based
duration model
Text analysis
Linguisticfeature

DNN-based
spectral andexcitation modelParameter generation
Acoustic feature
Vocoder decoding

Synthesized speech¢ Automatic construction of training corpus from audiobooks¢ Design of linguistic features for SPSS based on audiobooks¢ DNN-based SPSS

5

Automatic construction of training corpus¢ Mismatches are present in speech data and text

Substitution: misreading text
u Deletion: unrecording text
u Insertion: recording additional information, i.e., onomatopoeia⇒ This will negatively affect an acoustic model of SPSSu¢ Training corpus construction using speech recognizer

Texts are estimated from speech data [Braunschweiler, et al.; ’10]
u Texts may include speech recognition errors
uSpeech data
Correct text

hes came to a cottage knock knockBook text

hes came to a cottage

Recognized text

she came to a cottage knock knock

Speech recognition using a constrained word network is conducted6

Overview of training corpus construction
SI-SR part

SA-SR part

Audiobook database

Audiobook database

Speech data

Book text

Speech data

SI-SRDict.

SI-AM

SA-SR
BA-LM

Recognized text

SA-AM

BA-LM

Book andrecognized text

Pruning
Speechdata

Constraint SI-SR
SI-AM

Dict.

Recognized text

Pruning
Speechdata

Book text

Constraint network

Book andrecognized text
Constraint SI-SRSA-AM

Constraint network

Speech data and constraintrecognized text

Speech data and constraintrecognized text

Adaptation corpus

Training corpus for SPSS

SI: speaker-independent, SA: speaker-adapted, BA, book adapted,
SR: speech recognize, AM: acoustic model, LM: language model

7

Overview of training corpus construction
SI-SR part

SA-SR part

Audiobook database

Audiobook database

Speech data

Book text

Speech dataSI-SR
Dict.

SI-AM

SA-SR
BA-LM

Recognized text

SA-AM

BA-LM

Book andrecognized text

Pruning
Speechdata

Constraint SI-SR
SI-AM

Dict.

Recognized text

Pruning
Speechdata

Book text

Constraint network

Book andrecognized text
Constraint SI-SR

SA-AM

Constraint network

Speech data and constraintrecognized text

Speech data and constraintrecognized text

Adaptation corpus

Training corpus for SPSS

SI: speaker-independent, SA: speaker-adapted, BA, book adapted,
SR: speech recognize, AM: acoustic model, LM: language model8Adapted model¢ Language model (LM)Most book texts match speech datau LM based on book texts is useful for speech recognition
u Book-adapted LM is used for speech recognitionu¢ Acoustic model (AM)uSpeaker-adapted AM is constructed by using SI-SR results
SI-SR part

SA-SR part

Audiobook database

Audiobook databaseBook text

Book text
SI-SR
SI-AM

BA-LM

SA-SR
SA-AM

BA-LM

Adaptation corpus
9Overview of training corpus construction
SI-SR part

SA-SR part

Audiobook database

Audiobook database

Speech data

Book text

Speech data

SI-SRDict.

SI-AM

SA-SR
BA-LM

Recognized text

SA-AM

BA-LMBook andrecognized text

Pruning
Speechdata

Constraint SI-SR
SI-AM

Dict.

Recognized text

Pruning
SpeechdataBook text

Constraint network

Book andrecognized text
Constraint SI-SR

SA-AM

Constraint network

Speech data and constraintrecognized text

Speech data and constraintrecognized text

Adaptation corpus

Training corpus for SPSS

SI: speaker-independent, SA: speaker-adapted, BA, book adapted,
SR: speech recognize, AM: acoustic model, LM: language model

10Pruning¢ Word-match accuracy
uConcordance rate of book text and recognized text
Book text

hes came to a cottageword-match accuracy= 57%
Recognized text she came to a cottage knock knock
uLow word-match accuracy⇒ Reliability of text is lower¢ Pruning of low word match accuracy

If word-match accuracy is not more than threshold,speech data and text pair is pruned from training corpus
u Relation between quantity and quality of corpus is trade-off
uThreshold

Large

Small

Quantity of corpus

Small

Large

Quality of corpus

HighLow11

Overview of training corpus construction
SI-SR part

SA-SR part

Audiobook database

Audiobook database

Speech data

Book textSpeech data

SI-SR
Dict.

SI-AM

SA-SR
BA-LM

Recognized text

SA-AM

BA-LM

Book andrecognized text

Pruning
Speechdata

Constraint SI-SR
SI-AM

Dict.

Recognized text

Pruning
Speechdata

Book text

Constraint network

Book andrecognized text
Constraint SI-SR

SA-AM

Constraint network

Speech data and constraintrecognized text

Speech data and constraintrecognized text

Adaptation corpus

Training corpus for SPSS

SI: speaker-independent, SA: speaker-adapted, BA, book adapted,
SR: speech recognize, AM: acoustic model, LM: language model

12

Constraint speech recognition¢ Constraint-recognized text

Speech recognition using constrained word network consistingof book and recognized text
u Path penalty
ul Book text is NULL: path penalty for book textl Otherwise: path penalty for recognized text

uSpeech recognizer with constrained word network without LM
Book text

hes came to a cottage

Recognized text

Book text
Recognized text

heshe

she came to a cottage knock knock

cametoacottage

NULLknockknockContain text corresponding to additional speech informationu Reduce speech recognition errors
u13NITech system
Training corpus for SPSSSpeech

Text

Vocoder encoding

Text analysis

Acoustic features

Linguistic features

HMM training
Training part
Synthesis part
Input text

DNN training
Phoneme-levelalignment

HMM-basedduration model
Text analysis
Linguisticfeature

DNN-based
spectral andexcitation modelParameter generation
Acoustic feature
Vocoder decoding

Synthesized speech¢ Automatic construction of training corpus from audiobooks¢ Design of linguistic features for SPSS based on audiobooks¢ DNN-based SPSS14Design of linguistic features (1/2)¢ Linguistic features

Context-dependent model is used to capture contextual factors
u Appropriate linguistic features design is needed to synthesize
high-quality speechu Speech in conversational and descriptive parts of audiobook
ul Conversational part: emphatically, emotionally, etc.
l Descriptive part: comparatively neutrally⇒ Double quotes are used to express conversational partuProsodic informationl Intonation, rhythm, etc.

⇒ Detailed parsing results are used to express prosodic information¢ Linguistic feature using double quotes
uExample of added linguistic featuresl Whether the current phoneme is enclosed by double quotesl The rate of word enclosed by double quotes in this page

15

Design of linguistic features (2/2)¢ Linguistic feature using detailed parsing resultsuResults of parsing is represented by syntactic tree
Example ofsyntactic tree

ROOTSVPNP
DT

JJ

NN

VBZ.
NPNN

Input text:uThe

redpenismine.

Example of added linguistic featuresl Guess part-of-speech of the parent of the current wordl Distance on the syntactic tree between the current word and theprevious wordl Position of the current word in the parent of the current wordl The number of phonemes in the parent of the current word
16

NITech system
Training corpus for SPSSSpeech

Text

Vocoder encoding

Text analysis

Acoustic features

Linguistic features

HMM training
Training part
Synthesis part
Input text

DNN training
Phoneme-levelalignment

HMM-based
duration model
Text analysis
LinguisticfeatureDNN-based
spectral andexcitation modelParameter generation
Acoustic feature
Vocoder decoding

Synthesized speech¢ Automatic construction of training corpus from audiobooks¢ Design of linguistic features for SPSS based on audiobooks¢ DNN-based SPSS17

DNN-based SPSS¢ DNN-based SPSS [Zen, et al.; ’12]

DNN is trained to represent a mapping function from linguisticfeatures to acoustic featuresu DNN-based SPSS improves naturalness of synthesized speechu Inconsistency in training and synthesis criterionu Over-smoothing on speech parameter trajectories
uTrajectory training considering global variance18

DNN-based SPSS (1/3)¢ Frame-level trainingot

Frame-level likelihood

Synthesized speechc̄ = P W > ⌃G1 µForward-propagation

ltl : linguistic feature vectorµ : mean vector⌃G : globally tied covariance matrixo : speech parameter vectorµ1µ2・・・µT⌃G

Forward-propagation

⌃G・・・

µt・・・

Back-propagation

N (ot |µt , ⌃G )・・・
l1
l2lTc : static-feature vectorc̄ : optimal static-feature vector⇣⌘ 1>
1
P = W ⌃G W

W : window matrix

19

DNN-based SPSS (2/3)¢ Trajectory training [Hashimoto, et al.; ’16]

c1 c2 ・・・ cT
N (c|c̄, P )Trajectory likelihoodc̄1 c̄2 ・・・ c̄Tc̄ = P W > ⌃G1 µl1・・・

l2µT
Forward-propagationµ2・・・

Back-propagationµ1・・・

lT⌃Gl : linguistic feature vectorµ : mean vector⌃G : globally tied covariance matrix
c : static-feature vectorc̄ : optimal static-feature vector⇣⌘ 1>
1P = W ⌃G W

W : window matrix

20

DNN-based SPSS (3/3)¢ Trajectory training considering GV
GV likelihoodv(c)c1 c2 ・・・ cT[Hashimoto, et al.; ’16]

Trajectory likelihood

N (v(c)|v(c̄), ⌃GV )wT N (c|c̄, P )⌃GV v(c̄)

c̄1 c̄2 ・・・ c̄Tc̄ = P W > ⌃G1 µl1・・・

l2µT
Forward-propagationµ2・・・

Back-propagationµ1・・・lT⌃G

l : linguistic feature vectorµ : mean vector⌃G : globally tied covariance matrix
c : static-feature vectorc̄ : optimal static-feature vector⇣⌘ 1>
1P = W ⌃G W

W : window matrixv(·) : GV vector⌃GV : GV covariance matrix
21

Training corpus construction conditions
Children’s audiobook

50 books, 1090 pages

SR training corpus

WSJ0, WSJ1, TIMIT

Sampling rage

16 kHzFramewindow: Hamming, length: 25 ms, shift: 10 ms

Acoustic-feature

12-dimensional MFCC + Δ + ΔΔ

Acoustic model

3-state left-to-right GMM-HMM

Language model

tri-gram

Pruning threshold

word-match accuracy: 90%

22

TTS system conditions
Training corpus

825 pages (constraint-recognition text)

Sampling rage

44.1 kHzFrame

window: F0-adapteve Gaussian, shift: 5 ms

Acoustic feature

229-dimensional acoustic features(49-dimensional STRAIGHT mel-cepstrum,
24-dimensional aperiodicity measure + Δ + ΔΔ,
log F0, voiced/unvoiced features)

Linguistic feature

426-dimensional linguistic features(423-dimensional binary and numerical features,
three duration features)

HMM structure5-sate left-to-right MSD-HSMM

DNN structure

3 hidden layers with 2048 hidden units,
activation function: sigmoid,
dropout: 50%,
GV weight: 0.001

23

Experimental conditions of listening test
Participant

paid participants (104 native speakers)

Page domain

7 criteria,
60-point MOS

Sentence domain

2 criteria,
5-point MOSIntelligibility test

semantically unpredictable sentence (SUS),
word error rate (WER)

System17 systems (1 natural speech, 16 TTS systems)

24Speech samples¢ Automatic construction of training corpus from audiobooks

Text \ Threshold

80%

90%

100%

Book text
Recognized text
Constraintrecognized text
¢ Design of linguistic features for SPSS based on audiobooks

BaseDQParser

DQ + Parser

DNN

Trajectory DNN

Trajectory GV DNN¢ DNN-based SPSS

HMM

25

Experimental results¢ Page domain (60-point MOS)
Criterion

Overall
Pleasant
impression
nessSpeechpause

StressIntonation

Emotion

Listeningeffort

MOS

24

23

30

29

27

27

27

Rank

6th

7th

3rd

3rd

4th

5th

4th

¢ Sentence domain (5-point MOS)
Criterion

Naturalness

Similarity

MOS

3.0

2.6

Rank

5th

6th

¢ Intelligibility testWER

12%Rank

1st26

Experimental results¢ Page domain (60-point MOS)
Criterion

Overall
Pleasant
impression
nessSpeech
pauseStressIntonation

Emotion

Listeningeffort

MOS

24

23

30

29

27

27

27

Rank

6th

7th

3rd

3rd

4th

5th

4th

¢ Sentence domain (5-point MOS)
Criterion

Naturalness

Similarity

MOS

3.0

2.6

Rank

5th

6th

¢ Intelligibility test
WER

12%

Rank

1st

Page-level training and synthesis⇒ High MOS of speech pause

27Experimental results¢ Page domain (60-point MOS)
Criterion

Overall
Pleasant
impression
nessSpeechpause

StressIntonationEmotion

Listeningeffort

MOS

24

2330

2927

27

27

Rank

6th

7th

3rd

3rd

4th

5th

4th

¢ Sentence domain (5-point MOS)
Criterion

Naturalness

Similarity

MOS

3.0

2.6

Rank

5th

6th

¢ Intelligibility test
WER

12%

Rank

1st

Linguistic features of parsing
and trajectory training⇒ High MOS of stress and intonation
28

Experimental results¢ Page domain (60-point MOS)
Criterion

Overall
Pleasant
impression
nessSpeechpause

StressIntonation

Emotion

Listeningeffort

MOS

24

23

30

29

27

27

27

Rank

6th

7th

3rd

3rd

4th

5th

4th

¢ Sentence domain (5-point MOS)
Criterion

Naturalness

Similarity

MOS

3.0

2.6

Rank

5th

6th

¢ Intelligibility test
WER

12%Rank

1st

Training corpus include various
speaking style, emotion, character, etc.
⇒ Modeling is difficult⇒ Low MOS of similarity
29

Experimental results¢ Page domain (60-point MOS)
Criterion

Overall
Pleasant
impression
nessSpeechpause

StressIntonation

Emotion

Listeningeffort

MOS

24

23

30

29

27

27

27

Rank

6th

7th

3rd

3rd

4th

5th

4th

¢ Sentence domain (5-point MOS)
Criterion

Naturalness

Similarity

MOS

3.0

2.6

Rank

5th

6th¢ Intelligibility test
WER

12%Rank

1stLinguistic features of double quotes⇒ Can distinguish descriptive part⇒ Intelligible synthesized speech
30

Conclusion¢ TTS system developed for the Blizzard Challenge 2016
uNITech team focused on:l Automatic construction of training corpus from audiobooksl Design of linguistic features for SPSS based on audiobooksl DNN-based SPSSu

Large-scale subjective listening testsl Synthesized high natural and highest intelligible speechl Should improve speaker similarity¢ Future workuImproving linguistic featuresl Adding linguistic features of book, page, sentence, etc. codes

uImproving robustness of outliers
l ε-contaminated Gaussian loss [Zen, et al.; ’16]

31

32

Page-level training and synthesis¢ Explicit page-tuning sounds

Page-tuning sounds are not suited for training AM
u GMM is trained to detect page-tuning sounds
u Speech data are divided into page-by-page speech datau Page-level decoding, training, and synthesis are conductedu33

Design of linguistic features¢ Linguistic feature using page information

The number of {phrases, sentences} in this pageu position of the current sentence in this pageu¢ Linguistic feature using double quotes

Whether the {previous, current, next} {phoneme, syllable, word,phrase} is enclosed by double quotes
u The rate of {word, phrase} enclosed by double quotes in this pageu¢ Linguistic feature using detailed parsing results

Guess part-of-speech of the parent of the current word
u Distance on the syntactic tree between the current word and{the {previous, next} word, root of the syntactic tree, the {previous,next} content word}u Position of the current word in the parent of the current word
u The number of {phonemes, syllables, words} in the parent of thecurrent wordu34

HTS benchmark system¢ Text: INNOETICS + NII (shared)¢ Speech: page-level speech data (shared)¢ Differences from HTS STRAIGHT demo scripts

Page-level linguistic features (shared)u F0 extractor: RAPT, SWIPE’, PEAPER voting methodu Flat start using DAEM algorithm without phoneme alignmentu GV weight: 1.0 → 0.0001
u35