The Blizzard Machine
Learning Challenge 2017
Kei Sawada1, Keiichi Tokuda1,Simon King2, Alan W Black312

Nagoya Institute of Technology,

University of Edinburgh, 3Carnegie Mellon University

https://synsig.org/index.php/Blizzard_Challenge_2017
ASRU 2017 on December 18, 2017

Introduction¢ Text-to-speech (TTS) system

Technique for generating for artificial speech given input text
u Evaluation of methods for TTS systems
ul Comparisons are difficult when the training corpus,

Hello world

task, and listening test are different
TTS system¢ Blizzard Challenge [Black & Tokuda; ’05]

Better understand and compare research techniques in building
corpus-based TTS systems with the same datau A lot of time has to be spent on speech-specific tasks⇒ Not attractive to machine learning researchersu¢ Blizzard Machine Learning Challengeu

Focus on machine learning problems for speech synthesis
2History of TTS system
Rule-based

Corpus-based
Concatenative synthesis

Formant synthesis

Diphone synthesis
Unit selection synthesis
HMM/DNN-driven unit selection synthesis

Statistical speech synthesis
HMM-based speech synthesis
DNN-based speech synthesis

1980s

1990s

Heuristic approach

2000s

2010s
Statistical approach3

Statistical speech synthesis¢ Statistical speech synthesis
uMapping to speech waveform from text on
the basis of a statistical model¢ HMM-based speech synthesis (’95〜)

Context-dependent subword HMMs
u Regression trees to cluster and tie HMM statesu¢ DNN-based speech synthesis (’13〜)uHello world
Text analyzer
Linguisticfeatures
Acoustic model
Acousticfeatures
Vocoder

Replace regression trees with DNN¢ More recent DNN-based speech synthesis (’16〜)
uIntegration of vocoder and acoustic modelingl WaveNet, SampleRNN, etc.

uIntegration of text analyzer and acoustic modelingl Seq2seq model, Char2Wav, Tacotron, etc.

4Blizzard Challenge¢ Evaluations of TTS systems
uComparisons are difficult when the training corpus, task, andlistening test are different¢ Blizzard Challenge [Black, Tokuda, King, et al.]uGoall Better understand and compare research techniques in building corpus-

based TTS systems
l Evaluation campaign rather than competition⇒ Purpose of the challenge is to share knowledgeuMethodl Participants build voices on a common datasetl Organizers evaluate them in a single listening test

uAnnual Blizzard Challenge 2005-2017
l Need of construct all components for a complete TTS systeml A lot of time has to be spent on speech-specific tasks⇒ Not attractive to machine learning researchers
5

Blizzard Machine Learning Challenge 2017
¢ Blizzard Machine Learning Challenge

Does not involve speech-specific tasksu Allows participants to concentrate on machine learning problemu¢ Tasks

Text

Text

Text analyzer

Text analyzer

Linguisticfeatures

Linguisticfeatures

Text

Text analyzer+Acoustic model

Acoustic model
Acousticfeatures

Acoustic model+
Vocoder

Vocoder

Acousticfeatures

Text

Text analyzer+
Acoustic model+
Vocoder

Vocoder

Speech waveform

Speech waveform

2017-ES1

2017-ES2

Speech waveform

Speech waveform
6Tasks
Processes of the organizers

Processes of the participants¢ 2017-ES1
uPrediction of acoustic features from linguistic features

Training part

Synthesis part
DatasetTexts

Linguistic features

Speech waveforms

Acoustic features

Acousticmodel

Linguistic feature

Text

Acoustic feature

Speech waveform¢ 2017-ES2
uPrediction of speech waveforms from linguistic features

Training part

Synthesis part
DatasetTexts

Linguistic features
Speech waveforms

Acousticmodel

Linguistic featureText

Speech waveform7

Datasets (1/2)¢ DataCommercial-quality children’s audiobooks
from Usborne Publishing Ltd.
u Same as the Blizzard Challenge 2016
u 5 hours of speech datau"I'm king of the jungle," roared Lion.
"I'm going to eat you all up."
"No!" cried the jungle animals.

Character1
Character2
Descriptive part

¢ Data pruning

Mismatches between speech waveform and text
u Excessively expressive speech data (e.g. scream, singing voice)⇒ Negative effect on acoustic model trainingu Speech data including phoneme alignment errors were prunedu 4 hours of speech data (4651 files when divided into sentences)u8

Datasets (2/2)¢ Speech waveforms (2017-ES2)
u44.1kHz 16 bits monaural Waveform Audio File Format (WAVE)¢ Acoustic features (2017-ES1)
u77-dimensional acoustic featuresl Log F0 (linearly interpolated values in unvoiced parts)l Voiced and unvoiced informationl 50-dimensional mel-cepstrum representing spectral envelopel 25-dimensional mel-cepstrum representing aperiodicity measures¢ Linguistic features (2017-ES1 and 2017-ES2)
u687-dimentional linguistic featuresl Forced phoneme alignment ⇒ Frame-level linguistic featuresl Normalized to be within 0.0−1.0 based on minimum and maximum

9Systems¢ 7 teams registered and 3 teams submitted¢ Pairs of team ID and name are confidential

ID

Sampling Syn.
frequency speech

Category

Task

Model

A Natural speech−−44.1kHz

FFNN

44.1kHzX

Benchmark2017-ES1

YBenchmark

2017-ES1 FFNN + Trajectory training 44.1kHz

HSubmitted

2017-ES1

LSTM

44.1kHz

ISubmitted

2017-ES1LSTM + GAN postfilter

44.1kHz

ZBenchmark

2017-ES2

WaveNet16kHzGSubmitted

2017-ES2

LSTM + WaveNet

22.05kHz10

Listening test
¢ Design of listening test

The evaluation combined the entries for 2017-ES1 and 2017-ES2
into a single listening test
u 50 paid native listenersu¢ Evaluation criteriauNaturalness
l 5-point mean opinion score (MOS) test
l 1: completely unnatural – 5: completely naturalu

Speaker similarity
l 5-point MOS test
l 1: sounds like a different person – 5: sounds like the same person

uIntelligibilityl Dictation testl Word error rate (WER)l Semantically unpredictable sentence (SUS)

11

Good naturalness

Result (naturalness)
2017-ES1
2017-ES2

12

Good naturalness

Result (naturalness)
2017-ES1

Linguistic features

2017-ES2

LSTM
STFT

WaveNetOriginal WaveNet

Linguistic features

Linguistic features+Log F0

WaveNetWaveNetSpeech waveform

Speech waveform

Speech waveform

13

Good speaker similarity

Result (speaker similarity)
2017-ES1
2017-ES2

14

Good speaker similarity

Result (speaker similarity)
2017-ES1
2017-ES2

Linguistic features
LSTM
Acousticfeatures
GAN
Acousticfeatures
Vocoder
Speech waveform15

Result (intelligibility)
2017-ES1

Good intelligibility

2017-ES2

16

Result (intelligibility)
2017-ES1
2017-ES2

Linguistic features
WaveNetGood intelligibility

Speech waveform17

Discussion and future plan
¢ Recruit machine learning researchersuLack of advertisementl Difficult to control listening test if there are many participantsuQuality confirmation of synthesized speechl Release synthesized speech of benchmark system in advancel Release training script of benchmark system in advancel Release simple objective measureText¢ End-to-end speech synthesis

Text → Acoustic featureu Text → Speech waveform

Text analyzer

uLinguistic features
ES1

Acoustic model

ES3

Acoustic features
ES2

Vocoder

ES4

Speech waveform18Conclusions¢ Blizzard Machine Learning Challenge 2017u2017-ES1l Prediction of acoustic features from linguistic featuresu

2017-ES2l Prediction of speech waveform from linguistic features

uListening testl Naturalness, speaker similarity, and intelligibility evaluateduResults
l State-of-the-art machine learning approaches achieved higher scores

19