Statistical approach to speech synthesis---past, present, and future
Keiichi Tokuda
Nagoya Institute of Technology
INTERSPEECH 2019

Speech synthesis approaches• Rule-based, formant synthesis (~’90s)

Phonetic units are built by hand-crafted rules• Corpus-based, concatenative synthesis (’90s~)

Concatenate speech units (in acoustic feature or waveform) from a database• Single inventory: diphone synthesis• Multiple inventory: unit selection synthesis • Corpus-based, statistical synthesis (late ’90s~)
Source-filter model + statistical acoustic model• HMM (hidden Markov model) (1995~)>>• DNN (deep neural networks) (2013~)• WaveNet (2016~)

We wereworking on thisUnit-selection synthesis
Whole speech unit database
When we are lucky:
When we are unlucky:

Target cost
Concatenation cost
Selected speech units

・・・

・・・

Minimize the total cost in runtime
using dynamic programming

Statistical approach to speech synthesis
Text“Hello”
Text analysis

Controlling the
vocal organs“Hello”••••

Small footprint
Automatic voice building
Small language dependency
Flexibility/controllability“Hello”

HMM: hidden Markov model
DNN: deep neural network

Shape ofvocal tract

Speech“Hello”
Simulating speech production(digital filter / WaveNet vocoder)

Volume / Fundamental
frequency (+voicing)

Estimating thecontrol parameters

Represented bycontrol parameters

Air flow

Synthesized
speechPros and cons

Solved by neural vocoding, e.g., WaveNet

Unit selection
Waveform concatenation
 Natural sounding
Discontinuity, hit or missStatistical parametric
Vocoded  buzzy or muffled

SmoothWork better for larger databases

Can work for small databases

Large footprint

Small footprint

Fixed voice  fixed style,
fixed emotional expression, etc.

Flexible  speaker adaptation,
speaking style interpolation, etc.

No reason to hesitate to move onto the statistical approachOutline• Statistical formulation of speech synthesis• HMM-based speech synthesis• Deep neural networks• Evaluation / data & software tools• Other related topics

Outline• Statistical formulation of speech synthesis• HMM-based speech synthesis• Deep neural networks• Evaluation / data & software tools• Other related topics

The basic problem of speech synthesis
We have a speech database, i.e.,a set of pairs of texts and corresponding speech waveforms.
Given a text to be synthesized,
what is the speech waveform corresponding to the text?
• : texts• : speech waveforms

database• : text to be synthesized• : speech waveform

Given
?

Statistical formulation of speech synthesis (1/4)• Estimating predictive distribution is hard.
 Introduce generative representation ( : model parameters)• It is difficult to perform integral over auxiliary variables
 Approximate integral by maximizing
 training
 generation

Statistical formulation of speech synthesis (2/4)• Usually the generative model is decomposed into sub-modules, e.g.,

Feature extraction/waveform generation

Acoustic model: parametric representation of speech waveform: linguistic feature for: generative model parameter: acoustic model parameter: text analysis module parameter

Text analysis>>

Linguistic feature
Phoneme (or distinctive feature){preceding, current, succeeding} phonemes

Syllable












# of phonemes in {preceding, current, succeeding} syllable{accent, stress} of {preceding, current, succeeding} syllable
Position of current syllable in current word
# of {preceding, succeeding} {accented, stressed} syllable in current phrase# of syllables {from previous, to next} {accented, stressed} syllable
Vowel within current syllable, etc.

Word












Part of speech of {preceding, current, succeeding} word
# of syllables in {preceding, current, succeeding} word
Position of current word in current phrase# of {preceding, succeeding} content words in current phrase# of words {from previous, to next} content word
Syntactic/dependency information, etc.

Phrase# of syllables in {preceding, current, succeeding} phrase, etc.

()+Frame-level Duration and positional information+Speaking styles, emotional expressions, etc.
when we have such tags/lables

Statistical formulation of speech synthesis (3/4)• Decompose the generative model into sub-modules:
 training
 generation

 generation

training

Statistical formulation of speech synthesis (4/4)• It is difficult to perform integral and sum
 Approximated by step-by-step maximization: pre-trained text analysis module parameter speech feature parameter extraction
 labeling acoustic model training
 text analysis
 speech parameter generation
 waveform generation

Training
Synthesis

Outline• Statistical formulation of speech synthesis• HMM-based speech synthesis• Deep neural networks• Evaluation / data & software tools• Other related topics

HMM-based speech synthesis system
SPEECHDATABASE
Text analysis

Speech signal

Training part

Excitation
Parameter
extraction

SpectralParameter
Extraction

Excitation
parameters

Spectralparameters

Training HMMs

LabelsTEXT

Context-dependent HMMs
& state duration models

Text analysis
LabelsSynthesis part

Excitation
parameters

Parameter generationfrom HMMs

Excitation Excitation
generation

Spectralparameters

Synthesis
FilterSYNTHESIZEDSPEECHHMM-based speech synthesis system
SPEECHDATABASE

Speech signal

Training part
SpectralParameter
Extraction

Excitation
Parameter
extractionLˆ  arg max P( L | W , Λ )L
Text analysis

LabelsExcitation
parameters

Spectralparameters

Training HMMs

λˆ  arg max P(O | Lˆ , λ)λTEXToˆ  arg max P (o | lˆ, λˆ)Context-dependent HMMso

Text analysis

Parameter generationfrom HMMslˆ  argLabels
max PExcitation(l | w , Λ )parameterslSynthesis part& state duration models

Excitation Excitation
generation

Spectralparameters

Synthesis
FilterSYNTHESIZEDSPEECHHMM-based speech synthesis system
Speech signalLˆ  arg max P( L | W , Λ )SPEECHDATABASE
L

Text analysis

Excitation
Parameter
extraction

Training part
SpectralParameter
Extraction

Excitation
parameters

Spectralparameters

Training HMMs

LabelsTEXT

Context-dependent HMMs
& state duration models

Text analysis
LabelsExcitation
parameters

Parameter generationfrom HMMs

ˆ  arg max P(l | Excitation
Excitation
lSynthesis
w
,
Λ
)
part
l

generation

Spectralparameters

Synthesis
FilterSYNTHESIZEDSPEECH

17

HMM-based speech synthesis system
SPEECHDATABASE
Text analysis

Speech signal

Excitation
Parameter
extraction

Training part
SpectralParameter
Extraction

Excitation
parameters

Spectralparameters

Training HMMs

LabelsTEXT

Context-dependent HMMs
& state duration models

Text analysis
LabelsSynthesis part

Excitation
parameters

Parameter generationfrom HMMs

Excitation Excitation
generation

Spectralparameters

Synthesis
FilterSYNTHESIZEDSPEECHMel-cepstrum-based spectral analysis

ML estimation of mel-cepstrum:

when

is Gaussian process,is convex with respect to [Fuka92]

Warped frequency 𝜔 (rad) mel-cepstrum
2

warped frequency
mel-scale frequency

0

 /2



Frequency(rad)Spectral estimation example
Log Magnitude (dB)

100

50

0
0

4
Frequency (kHz)8

MLSA filter (1/2) [Fukada ’92]M

F ( z )   c ( m) ~
z m
m 0

Input













1  2

z 1



b(1)

b(m)  c(m)  b(m  1)

z 1

z 1



b(2)

z 1



b(3)




OutputMLSA filter (2/2) [Fukada ’92],,

• Approximation error < 0.24dB• O(8M) operations/sample• Stable filter


Input

F (z )






AL ,1

F (z )AL , 2F (z )
AL ,3

OutputL

F (z )AL , 4

exp F ( z ) 

1   AL ,l {F ( z )}ll 1L1   AL ,l { F ( z )}ll 1HMM-based speech synthesis system
SPEECHDATABASE
Text analysis

LabelsSpeech signal

Training part
SpectralParameter
Extraction

Excitation
Parameter
extraction

Excitation
parameters

Spectralparameters

Training HMMs

λˆ  arg max P (O | Lˆ , λ)λTEXTContext-dependent HMMs
& state duration models

Text analysis
LabelsSynthesis part

Excitation
parameters

Parameter generationfrom HMMs

Excitation Excitation
generation

Spectralparameters

Synthesis
FilterSYNTHESIZEDSPEECH

Hidden Markov model (HMM)
aij: state transition probabilitybq (ot ) : output probabilitya111b1 (ot )

a33

a22
a12

2

a23

b2 (ot )3b3 (ot )Observation
sequenceoo1 o2 o3 o4 o5  ・ ・ Statesequenceq

1 1 1 1 2  2 3

oT

 3

Structure of state output (observation) vector
Spectral parameters(e.g., mel-cepstrum, LSPs)
Spectrum part
log F0 with V/UV

Excitation part(e.g, F0)

Dynamic feature [Furui ’86]

Decision tree-based state clustering [Odell; ’95]
: HMM,: linguistic feature
L=voice?

k-a+b

yes

no

L=“w”?

t-a+h
R=silence?

…

…

s-i+n

yes

yes

R=silence?
no

yes

no
L=“gy”?

no

yes

no

C=unvoice?
C=vowel?
R=silence?
R=“cl”?
L=“gy”?
L=voice?

…

…

Sharing the parameter of HMMs in the same leaf node>>

Relation between two approaches
Unit Selection

HMM-based

ClusteredstatesMergedstates・・・ Sentence
Static

HMM

Delta

・・・

・
・
・


・・
・







Mean

Variancec

27

Stream-dependent tree-based clusteringState duration
model

Three dimensional Gaussian

HMM
Decision treesformel-cepstrum
Decision trees
for F0Decision tree forstate duration models

HMM-based speech synthesis system
SPEECHDATABASE
Text analysis

Speech signal

Text analysis
LabelsSynthesis part

SpectralParameter
Extraction

Excitation
Parameter
extraction

Excitation
parameters

Spectralparameters

Training HMMs

LabelsTEXT

Training partHMMsoˆ  arg max P (o | lˆ, λˆ)Context-dependent& state duration models
oExcitation
parameters

Parameter generationfrom HMMs

Excitation Excitation
generation

Spectralparameters

Synthesis
FilterSYNTHESIZEDSPEECHSpeech parameter generation algorithm: state sequenceqˆ  arg max P (q | lˆ, λˆ)  Determination of durationsqoˆ  arg max P (o | qˆ , λˆ)  Determination of speech parameteroGenerated speech parameter trajectory
/sil/

/a/

/i/

/sil/



32

Generated speech parameter trajectory
/sil/

/a/

/i/

/sil/



33

cq

Mean trajectory

Trajectory HMM
sil

a

i

d

a

sil
sil

with dynamic feature

5
10aw/o dynamic feature

15
20
25

i

30
d

35
40
45

a

50
55
sil

5

10

15

20

25

30

35

40

45

50

55

Time(frame)

Temporal covariance matrix

PqFlexibility to control speech variations• Speaker Adaptation (mimicking voices)• [Tamura ’98], [Tamura ’01], [Yamagishi ’03], …

()• Speaker Interpolation (mixing voices)• [Yoshimura ’97], …

• Eigenvoice (producing voices)• [Shichiri ’02], [Kazumi ’10], …

• Multiple-regression (controlling voices)• [Nose ’07], …

Only from publications by the HTS working groupMixing emotional expressions()Singing synthesis
HMM/DNN

1) HMM/DNN training
2) Singing generation
Score
withwithlyriclyric
Any
score

Synthesized singing

HMM+STRAIGHT

Outline• Statistical formulation of speech synthesis• HMM-based speech synthesis• Deep neural networks• Evaluation / data & software tools• Other related topics

Hidden Markov model approach
STRAIGHT

•
•
•
•
•
•
•

HMM

GV-based parameter generation [Toda ’05]
extraction model) [Zen ’07]
HSMM (hiddenFeaturesemi-Markov
Acoustic model/waveform generation
Trajectory HMM training [Zen ’07]
MGE training [Wu ’08]
Bayesian approach [Hashimoto ’09]
Additive decision tree [Takaki ’10]
Trainable excitation model [Maia ’07], etc.
Only from publications by the HTS working group

Text analysis

Recombining submodules

•
•
•
•

Joint estimation of acoustic Featureand excitation
models [Maia ’10]
extraction
Acoustic model/waveform generation
Log spectral distortion-version
of MGE training [Wu ’09]
Factor analyzed trajectory HMM (STAVOCO) [Toda ’08]
Mel-cepstral analysis-integrated HMM [Nakamura ’14]

Text analysis• Joint front-end / back-end training [Oura ’08]
Only from publications by the HTS working group

Deep neural network approaches (1/6)
FFNN, LSTM

Feature extraction/waveform generation

Acoustic model• DNN-based speech synthesis [Zen ’13]
• LSTM-based speech synthesis [Fan ’14], etc.

Text analysis

DNN vs HMM
DNN

HMM ( regression tree)• Work for larger database?
• Flat structure• Can work for small database?
• Plausible structure• Often prior knowledge / model
complexity is embedded ininitialization and/or training
process• Suitable for parallel/distributed
computation• Optimization in continuous space• Prior knowledge / model
complexity can be given in an
explicit form (e.g., model
structure)• Unsuitable for parallel/distributed
computation• Optimization in discrete space• Easy to implement• Difficult to shouting troubles• Difficult to implement• Easy to shouting troubles

Deep neural network approaches (2/6)
Sourcefilter model

FFNN, LSTM
>>

1. Measuring likelihoods
speech waveform directly,
Featureof
extraction
Acoustic model
Text analysis/waveform generation
2. train a neural network
3. which models both voiced and unvoiced components.
••

Directly modeling speech waveforms by neural networks [Tokuda ’15],
Directly modeling voiced and unvoiced components by neural networks [Tokuda ’16]

Speech signal model
Pulse train
Mixed phase

Voiced component

Speech signal
White Gaussianwith zero-mean and unit-variance

Minimum phase

Unvoiced component

Signal model for unvoiced+voiced sounds

Deep neural network approaches (3/6)
WaveNet, SampleRNN, WaveRNN, …
(autoregressive sturucture)

Feature extraction/waveform generation

Acoustic model

Text analysis

WaveNet• Autoregressive generative model using convolutional NN• Directly modeling speech waveform: waveform: acoustic and linguistic featuremodeled by using CNN• Dilated causal convolution

47

Speech signal generation model
𝑒𝑛

𝑥𝑛
𝑃 𝑧

𝑃 𝑧

Long-term predictor

Short-term predictor

𝑒𝑛



𝑥𝑛
WaveNetNonlinear Predictor

It works!

Famous words in speech technology (1980s)“Every time I fire a linguist,
the performance of the speech recognizer goes up”
by Frederick Jelinek“Every time I fire a speech technology researcher,the performance of the speech synthesizer goes up”
by ????? ?????

DNN variants for waveform modeling• Autoregressive• WaveNet, SampleRNN, WaveRNN, …

• Normalizing flow

• WaveGlow, Pallalel WaveNet, ClariNet, FloWaveNet, …

• Combining with source filter model• LPCNet, ExcitNet, GlotNet, LP-WaveNet, …

• Introducing signal processing technique• SubbandWaveNet, FFTNet, …

>>DNN vocoder with periodic excitation [Oura ’19]
• Autoencoder-type structure
extracts 3 dimensional periodic
signal• Decoder generates periodic
components and stochastic
components• WaveNet gives a constraints on
the intermediate variableDeep neural network approaches (4/6)
FFNN, LSTM

Feature extraction/waveform generation

Acoustic model

Text analysis• HSMM: duration model is included• FFNN, LSTM, WaveNet: external duration predictor is required

Frame-by-frame
conversion

Phoneme-level
From an external
duration predictor
Frame-level

It needs an external
duration predictor todetermine phone durations

Frame-level

DNN-HSMM
architecture

HSMM as the pdf of MDN

It runs at state-levelor phoneme-level[Tokuda ’16]
State-level orphoneme-level

Frame-level

Deep neural network approaches (5/6)
WaveNet vocoder(autoregressive structure)

Tacotron, Char2Wav, DeepVoice, …
(attention mechanism)>>Feature extraction/waveform generation

Acoustic model

Text analysis

Attention mechanism
Tacotron 2

Decoder

Acoustic feature
generation

Attention

mapping b/w
text and acoustic feature

Encoder

text analysis[from arXiv:1712.05884]

Deep neural network approaches (6/6)
WaveNet vocoder(autoregressive structure)

AttentionFeature extraction/waveform generation

Acoustic model

Text analysis

Controlling intermediate variables in thehierarchical structure• Language

High level• Japanese, English, Chinese, …

Text analysis• Dialect• Pronunciation• Pause• Allophone• Prosody• Accent, stress, tone, …

• Speaking style, emotional expression• Emphasis• Nonverbal, paralinguistic information• Voice characteristics• Male, female, child, adult, eldarly• Speech parameter• Fundamental frequency, volume, duration, aperiodic component, …

Acoustic modelLow level

Vocoding

Singing synthesis with CNN+WaveNet
WaveNet vocoder

CNN

Feature extraction/waveform generation

Acoustic model

Text analysis

Other DNN techniques and architectures• GAN• VAE/VQ-VAE• Transformer (self attention)

Outline• Statistical formulation of speech synthesis• HMM-based speech synthesis• Deep neural networks• Evaluation / data & software tools• Other related topics

Blizzard Challenge• Performance of TTS system depends on the database• Difficult to compare techniques themselves“Blizzard Challenge”

Evaluating corpus-based speech synthesison common datasets [Black ’05]
Since 2005

Evaluation methodology• Naturalness• Mean Opinion Score• Speaker similarity• Degradation Mean Opinion Score• Intelligibility (dictation of SUS, PCS, etc.)• Word accuracy

Not enough for spontaneous speech, audio book task, etc.

Common datasets for speech synthesis• ELRA http://www.elra.info/• ELDA http://www.elda.org/• LDC https://www.ldc.upenn.edu/• OpenSLR http://www.openslr.org/• ARCTIC• VCTK• LibriTTS, …
Not so many because it needs studio-quality recordings

Software tools• ISCA SynSig https://www.synsig.org/index.php/Software• ISCA SCOOT https://www.isca-speech.org/iscaweb/index.php/scootSoftware tools
Toolkit for building voice interaction systems
Speech recognition engine

Speech synthesis engine

Japanese TTS system

Training toolkit

Singing synthesis system

Speech signal processing toolkit

Takashi Masuko, Noboru Miyazaki, Kazuhito Koishida, Takayoshi Yoshimura,
Heiga Zen, Junichi Yamagishi, Keiichiro Oura, Akinobu Lee and others contributed

Outline• Statistical formulation of speech synthesis• HMM-based speech synthesis• Deep neural networks• Evaluation / data & software tools• Other related topics• Text normalization• Voce conversion• Speech coding• Anti-spoofing• Physical simulation

Text normalization• Text normalization is excluded from the end-to-end systems• Still rule-based approach is the mainstream• It would be included in the end-to-end process in the near future

Voice conversion• Close relationship to speech synthesis• DNN-approach has emerged also in voice conversion research• Realtime application is essential• Realtime (or low-latency) prosody conversion is a challenging
problemSpeech coding• WaveNet and other waveform modeling approaches seems tobring a revolution to speech coding.
•
•
•
•
•
•

WaveNet based low rate speech coding [Kleijn ’18]
A Real-Time Wideband Neural Vocoder at 1.6 kb/s Using LPCNet [Valin ’19]
Low Bit-rate Speech Coding with VQ-VAE and WaveNet [Garbacea ’19]
High-quality speech coding with sample RNN [Klejsa ’19]
WaveNet-based zero-delay lossless speech coding [Yoshimura ’18]
Wavenet-based delay-free ADPCM Speech Coding [Yoshimura ’19]

Imposture using speech synthesis• Fear for spoofing with speech synthesis• On the security of HMM-based speaker verification systems against
imposture using synthetic speech [Masuko ’99]

• Detecting synthesized speech• A robust speaker verification system against imposture using an HMMbased speech synthesis system [Satoh ’01]

• ASVspoof 2015
• The First Automatic Speaker Verification Spoofing and Countermeasures
Challenge

Physical simulation vs Deep neural network• In the future, techniques for measuring dynamics of vocal tract will
be significantly progressed.
• Also, techniques for simulating speech production system will be
progressed.
will it be possible to generate natural-sounding speechbased on the physical simulation approach?
• Advantage: realistic constraints, lower dimensional representation
 latent representation in DNN-based system?

Summary
Statistical approach to speech synthesis• Now it has reached at the level that we cannot tell the difference
between human and machine• Still we have a lot of problems to be solved • More flexibility and controllability for realizing diversity of speech

Let us enjoy speech synthesis research!
Thank you!

>>

Speech synthesis in the future• Spoken dialog system

Cross-lingual/multilingual• Speech
Lettranslation
us enjoy speech synthesis
research
Cross-media

for realizing diversity of speech!

• Support for people with disabilities• CALL• Content creation

Flexibility/diversity

Low
resource
languageThank you!
Editor design

Common data

Special thanks• Supervisors: Satoshi Imai, Tadashi Kitamura, Takao Kobayashi• Colleagues & students: Takashi Masuko, Noboru Miyazaki, Takayoshi Yoshimura,
Shinji Sako, Masatsune Tamura, Junichi Yamagishi, Tomoki Toda, Heiga Zen, Kazuhito
Koishida, Tetsuya Yamada, Nobuaki Mizutani, Ryuta Terashima, Akinobu Lee, Keiichiro
Oura, Keijiro Saino, Kenichi Nakamura, Yi-Jian Wu, Ling-Hui Chen, Shifeng Pan,
Yoshihiko Nankaku , Ranniery Maia, Sayaka Shiota, Chiyomi Miyajima, Kei Hashimoto,Shinji Takaki, Kazuhiro Nakamura, Kei Sawada, Takenori Yoshimura, Daisuke
Yamamoto, …
• Collaborators and advisors: Junichi Takami, Naoto Iwahashi, Mike Schuster, Satoshi
Nakamura, Frank Soong, Mchiael Picheny, Simon King, Steve Young, Mari Ostendorf,
Alan Black, Alex Acero, Bill Byrne, Phil Woodland, Thomas Hain, Phil Garner, Masataka
Goto, Shigeru Katagiri, Hideki Kenmochi, Kazuya Takeda, Tatsuya Kawahara, Sadaoki
Furui, Seiichi Nakagawa, Keikichi Hirose, Tetsunori Kobayashi, Miko Kurimo, Shigeki
Sagayama, Kiyohiro Shikano, Hisashi Kawai, Nobuyuki Nishizawa, Minoru Tsuzaki,
Yoichi Yamashita, Nobuaki Minematsu, Mat Shannon, Mark Gales, Kai Yu, John Dines,…
in random order. I am sorry but I may have missed many…

References• K. Tokuda, T. Kobayashi, S. Shiomoto, S. Imai, “Adaptive filtering based on
cepstral representation —adaptive cepstral analysis of speech,” ICASSP 1990a.
• K. Tokuda, T. Kobayashi, S. Imai, “Generalized cepstral analysis of speech —unified approach to LPC and cepstral method,” ISCSLP 1990b.
• T. Fukada, K. Tokuda, T. Kobayashi, S. Imai, “An adaptive algorithm for melcepstral analysis of speech,” ICSSP 1992.
• K. Tokuda, T. Kobayashi, T. Masuko, S. Imai, “Mel-generalized cepstral analysis—a unified approach to speech spectral estimation,” ICSLP 1994.
• K. Tokuda, T. Kobayashi, S. Imai, “Speech parameter generation from HMM
using dynamic features,” ICASSP 1995a.
• K. Tokuda, T. Masuko, T. Yamada, T. Kobayashi, S. Imai, “An algorithm forspeech parameter geeration from continuous mixture HMMs with dynamic
features,” EUROSPEECH 1995b.
Only from HTS group• T. Masuko, K. Tokuda, T. Kobayashi, S. Imai, “HMM-based speech synthesis
with various voice characteristics,” ASA/ASJ Joint Meeting 1996.
• T. Masuko, K. Tokuda, T. Kobayashi, S Imai, “Voice characteristics conversion
for HMM-based speech synthesis system,” ICASSP 1997.
• T. Yoshimura, T. Masuko, K. Tokuda, T. Kobayashi, T. Kitamura, “Speaker
interpolation in HMM-based speech synthesis system,” EUROSPEECH 1997.
• T. Masuko, T. Kobayashi, M. Tamura, J. Masubuchi, K. Tokuda, “Text-to-visual
speech synthesis based on parameter generation from HMM,” ICASSP 1998.
• M. Tamura, T. Masuko, K. Tokuda, T. Kobayashi, ”Speaker adaptation for
HMM-based speech synthesis system using MLLR,” SSW 1998.
• T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, T. Kitamura, “Simultaneousmodeling of spectrum, pitch and duration in HMM-based speech synthesis,”EUROSPEECH 1999.
• K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, T. Kitamura, “Speechparameter generation algorithms for HMM-based speech synthesis,” ICASSP
2000.
SSW: ISCA Speech Synthesis Workshop• T. Masuko, T. Hitotsumatsu, K. Tokuda, T. Kobayashi, “On the security of
HMM-based speaker verification systems against imposture using synthetic
speech,” EUROSPEECH 1999.
• T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, T. Kitamura, “Simultaneousmodeling of spectrum, pitch and duration in HMM-based speech synthesis,”EUROSPEECH 1999.
• T. Masuko, K. Tokuda, T. Kobayashi, “Imposture using synthetic speech against
speaker verification based on spectrum and pitch,” ICSLP/INTERSPEECH 2000.
• S. Sako, K. Tokuda, T. Masuko, T. Kobayashi, T. Kitamura, “HMM-based text-toaudio-visual speech synthesis,” ICSLP/INTERSPEECH 2000.
• M. Tamura, T. Masuko, K. Tokuda, T. Kobayashi, “Text-to-speech synthesis with
arbitrary speaker’s voice from average voice,” EUROSPEECH 2001.
• T. Satoh, T. Masuko, T. Kobayashi, K. Tokuda, “A robust speaker verification
system against imposture using an HMM-based speech synthesis system,”EUROSPEECH 2001.
• K. Shichiri, A. Sawabe, K. Tokuda, T. Masuko, T. Kobayashi, T. Kitamura,“Eigenvoices for HMM-based speech synthesis,” ICSLP 2002.

• J. Yamagishi, T. Masuko, K. Tokuda, T. Kobayashi, “A training method for
average voice model based on shared decision tree context clustering andspeaker adaptive training,” ICASSP 2003.
• K. Tokuda, H. Zen, T. Kitamura, “Trajectory modeling based on HMMs with theexplicit relationship between static and dynamic features,” EUROSPEECH 2003.
• H. Zen, K. Tokuda, T. Masuko, T. Kobayashi, T. Kitamura, “Hidden semi-Markov
model based speech synthesis,” ICSLP 2004.
• T. Toda, A. Black, K. Tokuda, “Spectral conversion based on maximum
likelihood estimation considering global variance of converted parameter,”
ICASSP 2005.
• T. Toda, K. Tokuda, “Speech parameter generation algorithm considering
global variance for HMM-based speech synthesis,” INTERSPEECH 2005.
• A. Black, K. Tokuda, “The Blizzard Challenge – 2005: Evaluating corpus-based
speech synthesis on common datasets,” INTERSPEECH 2005.
• K. Saino, H. Zen, Y. Nankaku, A. Lee, K. Tokuda, “HMM-based singing voicesynthesis system,” Interspeech 2006.

• R. Maia, T. Toda, H. Zen, Y. Nankaku, K. Tokuda, “An excitation model for
HMM-based speech synthesis based on residual modeling,” SSW 2007.
• K. Oura, Y. Nankaku, T. Toda, K. Tokuda, R. Maia, S. Sakai, S. Nakamura,“Simultaneous Acoustic, Prosodic, and Phrasing Model Training for TTS
Conversion Systems,” ISCSLP 2008.
• K. Hashimoto, H. Zen, Y. Nankaku, K. Tokuda, “A Bayesian approach to HMMbased speech synthesis,” ICASSP 2009.
• K. Kazumi, Y. Nankaku, K. Tokuda, “Factor analyzed voice models for HMMbased speech synthesis,” ICASSP 2010.
• K. Oura, A. Mase, T. Yamada, S. Muto, Y. Nankaku, K. Tokuda, “Recent
Development of the HMM-based Singing Voice Synthesis System – Sinsy,” SSW
2010.
• K. Nakamura, K. Hashimoto, Y. Nankaku, K. Tokuda, “Integration of Acoustic
Modeling and Mel-cepstral analysis for HMM-based Speech Synthesis,” ICASSP
2013.
• K. Tokuda, H. Zen, ``Directly modeling voiced and unvoiced components inspeech waveforms by neural networks,'' ICASSP 2016.

• K. Tokuda, K. Hashimoto, K. Oura, Y. Nankaku, “Temporal modeling in neuralnetwork based statistical parametric speech synthesis,” SSW 2016.
• T. Yoshimura, K. Hashimoto, K. Oura, Y. Nankaku, K. Tokuda, “Mel-cepstrumbased quantization noise shaping applied to neural-network-based speech
waveform synthesis,” IEEE/ACM Transactions on Audio, Speech, and Language
Processing, vol. 26, no. 7, pp. 1173-1180, July, 2018.
• J. Niwa, T. Yoshimura, K. Hashimoto, K. Oura, Y. Nankaku, K. Tokuda,“Statistical voice conversion based on WaveNet,” ICASSP 2018.
• T. Yoshimura, K. Hashimoto, K. Oura, Y. Nankaku, K. Tokuda, “WaveNet-based
zero-delay lossless speech coding,” SLT 2018.
• T Yoshimura, K. Hashimoto, K. Oura, Y. Nankaku, K. Tokuda, ``Speakerdependent WaveNet-based delay-free ADPCM speech coding,'' ICASSP 2019.
• S. Takaki, Y. Nankaku, K. Tokuda, “Spectral modeling with contextual additive
structure for HMM-based speech synthesis,” SSW 2010.