Flexible speech synthesis basedon hidden Markov models
Keiichi Tokuda
Nagoya Institute of Technology

APSIPA ASC 2013, Kaohsiung
November 1, 2013

Towards human-like talking machines
For realizing natural human-computer interaction,
speech synthesis systems are required to have an
ability to generate speech with:– arbitrary speaker’s voice– speaking styles(e.g., reading, conversational)– emotional expressions(e.g., happy, angry, sad)
– emphasis– and so on

1

History
Rule-based, formant synthesis (~’90s)– Hand-crafting each phonetic units by rules

Corpus-based, concatenative synthesis (’90s~)– Concatenate speech units (waveform) from a database• Single inventory: diphone synthesis• Multiple inventory: unit selection synthesis

Corpus-based, statistical parametric synthesis– Source-filter model + statistical acoustic model• Hidden Markov model: HMM-based synthesis
2General unit-selection synthesis scheme
Whole speech unit database

Selected speech units

・・・

・・・

Target cost

Concatenation cost3Overview of this talk
1. Introduction and background
2. Basic techniques in the system
3. Examples demonstrating its flexibility
4. Discussion and conclusion4

HMM-based speech synthesis system
SPEECHDATABASE
Text analysis

Speech signal

Excitation
Parameter
extraction

Excitation
parameters

Training part
SpectralParameter
Extraction

Training HMMs

LabelsTEXT

Context-dependent HMMs
& state duration models

Text analysis
LabelsSynthesis part

Spectralparameters

Excitation
parameters

Parameter generationfrom HMMs

Excitation Excitation
generation

Spectralparameters

Synthesis
FilterSYNTHESIZEDSPEECH
5

HMM-based speech synthesis system
SPEECHDATABASE
Text analysis

Speech signal

Excitation
Parameter
extraction

Excitation
parameters

Training part
SpectralParameter
Extraction

Training HMMs

LabelsTEXT

Context-dependent HMMs
& state duration models

Text analysis
LabelsSynthesis part

Spectralparameters

Excitation
parameters

Parameter generationfrom HMMs

Excitation Excitation
generation

Spectralparameters

Synthesis
FilterSYNTHESIZEDSPEECH6

Human speech production

Modulation of carrier wave
by speech information
Frequencytransfercharacteristics

SpeechSound source
Voiced: pulse
Unvoiced: noise

Magnitude
start--end
Fundamental
frequency
air flow7

Source-filter model

Source excitation part
Pulse train

White noise

Excitatione(n)Vocal tract resonance part
Lineartime-invariant
systemH (z )

Speechx ( n ) = h ( n ) * e( n )8ML estimation of spectral parameter−−mm~
H ( z ) = exp
exp∑
c
(
m
)
z
(
)
c
m
z
∑

m==00
m

z −α
− jω~
−1
~
z =
=e
−1
1−α z
−1

Warped frequency

M
M

(rad)

Mel-cepstral representation of speech spectraππ2warped frequency
mel-scale frequency

0

π /2

Frequency ω (rad)πML-estimation of mel-cepstrumc = arg max p ( x | c )cx : short segment of speech waveformc : mel-cepstrum(Gaussian process)9Synthesis FilterM−m~
H ( z ) = exp ∑ c(m) z
m =0M

F ( z ) = ∑ c ( m) ~
z −m
m =0

Input

−

α

+

−

α

+

α

1−α 2

z −1

+

b(1)

b(m) = c(m) − αb(m + 1)

z −1

z −1

+

b(2)

z −1

+

b(3)
+

+

Output10

Structure of MLSA filterL

H ( z ) = exp F ( z ) ≅1 + ∑ AL ,l {F ( z )}l
l =1L

1 + ∑ AL ,l {− F ( z )}l
l =1

+
Input

+
−

−

F (z )F (z )F (z )F (z )

AL ,1

AL , 2AL ,3AL , 4

Output11

Waveform reconstruction
Original speech

F0

Unvoiced / voiced

Mel-cepstrum

Pulse train

Synthesis filter
Excitation
White noisee(n)H (z )

reconstructed speech

These speech parameters have to be estimatedfrom the text to be synthesized  use of HMM

x(n)

12

HMM-based speech synthesis system
SPEECHDATABASE
Text analysis

Speech signal

Excitation
Parameter
extraction

Excitation
parameters

Training part
SpectralParameter
Extraction

Training HMMs

LabelsTEXT

Context-dependent HMMs
& state duration models

Text analysis
LabelsSynthesis part

Spectralparameters

Excitation
parameters

Parameter generationfrom HMMs

Excitation Excitation
generation

Spectralparameters

Synthesis
FilterSYNTHESIZEDSPEECH13

Hidden Markov model (HMM)
aij: state transition probabilitybq (ot ) : output probabilitya111b1 (ot )

a33

a22
a12

2

a23

b2 (ot )3b3 (ot )Observation
sequenceoo1 o2 o3 o4 o5  ・ ・ Statesequenceq

1 1 1 1 2  2 3

oT

 3
14

Structure of state output (observation) vector
Spectral parameters(e.g., mel-cepstrum, LSPs)
Spectrum part∆∆∆log F0 with V/UV

Excitation part

∆
∆∆
15

Observation of F0
unvoicedΩ2 = R0

voicedΩ1 = R1

Log Frequency

・

Time

Unable to model by continuous or discrete distributions⇒ Multi-space distribution HMM (MSD-HMM)
16

Multi-space probability distribution HMM(MSD-HMM)

1Ω1 = Rn1Ω2 = R

n2

w2,1

w1,1

N11 ( x )

ΩM = RnM

w3,1

N 21 ( x )
w2, 2

w1, 2

3

2

N 31 ( x )w3, 2

N12 ( x )N 22 ( x )N 32 ( x )







w1, M
N1 M ( x )w2, M

N 2M ( x)w3, M

N 3M ( x )17

MSD-HMM for F0 modeling1

HMM for F0Ω1 = R1Ω2 = R

w2,1

w1,1
Voiced

0

w1, 2

・

Unvoiced

3

2
w3,1

Voicedw2, 2

・

Unvoiced

Voicedw3, 2・

Unvoiced

Voiced / Unvoiced weights
18

Contextual factors
Phoneme{preceding, succeeding} two phonemes
current phoneme

Syllable












# of phonemes in {preceding, current, succeeding} syllable{accent, stress} of {preceding, current, succeeding} syllable
Position of current syllable in current word
# of {preceding, succeeding} {accented, stressed} syllable in current phrase# of syllables {from previous, to next} {accented, stressed} syllable
Vowel within current syllable

Word










Part of speech of {preceding, current, succeeding} word
# of syllables in {preceding, current, succeeding} word
Position of current word in current phrase# of {preceding, succeeding} content words in current phrase# of words {from previous, to next} content word

Phrase# of syllables in {preceding, current, succeeding} phrase…..

Huge # of combinations ⇒ Difficult to have all possible models
 19

Decision tree-based state clustering [Odell; ’95]

L=voice?

k-a+b

yes
L=“w”?

t-a+h
R=silence?

…

…

s-i+n

no

…

C=unvoice?
C=vowel?
R=silence?
R=“cl”?
L=“gy”?
L=voice?

yes

yes

R=silence?
no

yes

no
L=“gy”?

no

yes

no

…

Sharing the parameter of HMMs in the same leaf node20

Stream-dependent tree-based clustering (2)State duration
model

Three dimensional Gaussian

HMM
Decision treesformel-cepstrum

Decision tree
for state dur.
models

Decision trees
for F021

HMM-based speech synthesis system
SPEECHDATABASE
Text analysis

Speech signal

Excitation
Parameter
extraction

Excitation
parameters

Training part
SpectralParameter
Extraction

Training HMMs

LabelsTEXT

Context-dependent HMMs
& state duration models

Text analysis
LabelsSynthesis part

Spectralparameters

Excitation
parameters

Parameter generationfrom HMMs

Excitation Excitation
generation

Spectralparameters

Synthesis
FilterSYNTHESIZEDSPEECH22

Speech parameter generation algorithm [Tokuda; ’00]
For given sentence HMM, determine a speech parameter
T
T
T T
o
=
[
o
,
o
,

,
o
vector sequence
1
2
T ] which maximizes
P (o | lˆ, λˆ) = ∑ P (o | q, λˆ)P (q | lˆ, λˆ)q≈ max P (o | q, λˆ) P (q | lˆ, λˆ)q⇒qˆ = arg max P(q | lˆ, λˆ)qoˆ = arg max P(o | qˆ , λˆ)o23Determination of state sequence (1/3)
aij

a11

: State transition probabilitybq (ot ) : Output probability1b1 (ot )Observation
sequence
State sequence
State duration

a33

a22
a12

2

a23

b2 (ot )3b3 (ot )o o1 o2 o3 o4 o5  ・ ・  oT
q 1 1 1 1 2  2 3

 3

d

5

4

10

Determine state sequence via determining state durations
24

Determination of state sequenceK

P (q | lˆ, λˆ) = ∏ pi (d i )i =1

pi (⋅) : state-duration distribution of i -th stated i : state duration of i -th state

K : # of states in a sentence HMM for lˆGaussian(pi (d i ) = N d i | mi , σ2i) ⇒ dˆ = mii25Speech parameter generation algorithm
For given HMM λ ,determine a speech parameter vector
T
T
T T
o
=
[
o
,
o
,

,
o
Sequence
1
2
T ] which maximizes
P (o | lˆ, λˆ) = ∑ P (o | q, λˆ)P (q | lˆ, λˆ)q≈ max P (o | q, λˆ) P (q | lˆ, λˆ)q⇒qˆ = arg max P(q | lˆ, λˆ)qoˆ = arg max P(o | qˆ , λˆ)o26Without dynamic feature

becomes a sequence of mean vectors⇒ discontinuous outputs between states27

Dynamic features

∂ct
∆ct =
≈ 0.5(ct +1 − ct −1 )
∂t
2
∂ ct
2
∆ ct = 2 ≈ ct +1 − 2ct + ct −1
∂t
ct +1

ct −1

− 0.5

0.5

∆ct −1∆ct +1

ct −1

ct
∆ct

1

ct
−2

ct +1
1

∆ct −1∆2 ct∆ct +128

Integration of dynamic features
Relationship between speech parameter vectors &static feature vectors
o
c
W

[

]

T
t

T
t

2 T T
t

M

M

M

ot = c , ∆c , ∆ c
3M

 29

Solution for the problem (1/2)
By setting

we obtain

Wc∂ log P(O | qˆ , λ )= O,∂cW T Σ q−ˆ 1Wc = W T Σ q−ˆ 1 μqˆ ,wherec = [c1T , c2T ,  , cTT ]T
μqˆ = [ μ , μ ,  , μ ]
T
qˆ1

T
qˆ 2

T T
qˆT

Σ qˆ = [ Σ , Σ ,  , Σ ]
T
qˆ1

T
qˆ 2

T T
qˆT
30

Solution for the problem (2/2)
W

Σ

T

=

−1
qˆ

W

W

T

c

Σ q−ˆ 1μqˆ

31Generated speech parameter trajectory 32

Generated spectra
sil

a

i

sil

0 1 2 3 4 5(kHz)

w/o dyn0 1 2 3 4 5
(kHz)

w dyn

Spectra changing smoothly at state boundaries

33Frequency

Generated F0natural speech
200
150
100
50

Frequency

02time (s)without dynamic features
200
150
100
50
0

Frequency

1

1

2

time (s)with dynamic features
200
150
100
50
0

1

2

time (s)34Effect of dynamic features
Mel-cepstrum
static+2∆+∆

staticstatic+2log F0∆+∆

Smooth!

static35

HMM-based speech synthesis system
SPEECHDATABASE
Text analysis

Speech signal

Excitation
Parameter
extraction

Excitation
parameters

Training part
SpectralParameter
Extraction

Training HMMs

LabelsTEXT

Context-dependent HMMs
& state duration models

Text analysis
LabelsSynthesis part

Spectralparameters

Excitation
parameters

Parameter generationfrom HMMs

Excitation Excitation
generation

Spectralparameters

Synthesis
FilterSYNTHESIZEDSPEECH36

Overview of this talk
1. Introduction and background
2. Basic techniques in the system
3. Examples demonstrating its flexibility
4. Discussion and conclusion

37

Emotional speech synthesis
textneutralangry「授業中に携帯いじってんじゃねえよ！
電源切っとけ！」“Don’t touch your cell phone during aclass! Turn off it!”
「ミーティングには毎週参加しなさい！」“You must attend the weekly meeting!”

trained with 200 utterances

Speaker adaptation (mimicking voices)
MLLR-based adaptation
Adaptationdata
speaker A

Speaker-independent

adaptation

Adapted model

w/o adaptation (initial model)
Adapted with 4 utterances
Adapted with 50 utterances
Speaker A’s speaker-dependent system？

Speaker interpolation (mixing voices)
Linear combination of two speaker-dependent models

Model A

Model B
Interpolated model

A: 1.00

0.75

0.50

0.25

0.00

B: 0.00

0.25

0.50

0.75

1.00

Voice morphing
Two voices:Α  ΒΑ  Β

Four voices:Interpolation of speaking styles

Base model A

Base model B

Interpolation

Neutralextrapolation

High Tension

Eigenvoice (creating voices) [Shichiri; ’02]
Speaker 1

Speaker 2

μ

e (1)

Speaker S

e (k )e( K )
System Overview
Click here
for a demoMultilingual speech synthesis
•
•
•
•
•
•
•
•
•
•
•
•

Japanese
American English
Chinese (Mandarin) (by ATR)
Brazilian Portuguese (by Nitech, and ＵＦＲＪ)
European Portuguese (by Nitech, Univ of Porto, and UFRJ)
Slovenian(by Bostjan Vesnicer, University of Ljubljana, Slovenia)
Swedish (by Anders Lundgren, KTH, Sweden)
German (by University of Bonn, and Nitech)
Korean (by Sang-Jin Kim, ETRI, Korea)
Finish (by TKK, Finland)
Baby English (by Univ of Edinburgh, UK)
Polish, Slovak, Arabic, Farsi, Croatian, Polyglot, etc.



HMM-based singing synthesis

HMM1) HMM training

Score
with
lylic
Any
score
with
lylic

2) Singing generation
synthehsized siging

45

Flexibility in singing synthesis• Sing a popular song

• Sing by a famous person’s voice(from a TV program NHK “Science zero”)• Rap singing

46

MMDAgent
A toolkit for building voice interaction systems••••

Fully open-source toolkit with open interfaces
Tightly integrated speech recognition/synthesis engines
3-D scene rendering fully compatible with CG tools
HMM-based flexible and expressive speech synthesis(neutral, angry, bashful, happy, sad)47

Summary
Statistical approach to speech synthesis• Whole speech synthesis process is described in a
unified statistical framework• It can provide flexibility: various voices, emotional
expressions, speaking styles, etc.
Future work
• Still we have many problems should be solved:• Direct modeling of speech waveform• Importance of the database48

References (1/4)
Sagisaka;'92 - "ATR nu-TALK speech synthesis system," ICSLP, '92.
Black;'96 - "Automatically clustering similar units...," Euro speech, '97.
Beutnagel;'99 - "The AT&T Next-Gen TTS system," Joint ASA, EAA, & DAEA meeting, '99.
Yoshimura;'99 - "Simultaneous modeling of spectrum ...," Eurospeech, '99.
Itakura;'70 - "A statistical method for estimation of speech spectral density...," Trans. IEICE, J53-A, '70.
Imai;'88 - "Unbiased estimator of log spectrum and its application to speech signal...," EURASIP, '88.
Kobayashi;'84 - "Spectral analysis using generalized cepstrum," IEEE Trans. ASSP, 32, '84.
Tokuda;'94 - "Mel-generalized cepstral analysis -- A unified approach to speech spectral...," ICSLP, '94.
Imai;'83 - "Cepstral analysis synthesis on the mel frequency scale," ICASSP, '83.
Fukada;'92 - "An adaptive algorithm for mel-cepstral analysis of speech," ICASSP, '92.
Itakura;'75 - "Line spectrum representation of linear predictive coefficients of speech...," J. ASA (57), '75.
Tokuda;'02 - "Multi-space probability distribution HMM," IEICE Trans. E85-D(3), '02.
Odell;'95 - "The use of context in large vocaburary...," PhD thesis, University of Cambridge, '95.
Shinoda;'00 - "MDL-based context-dependent subword modeling...," Journal of ASJ(E) 21(2), '00.
Yoshimura;'98 - "Duration modeling for HMM-based speech synthesis," ICSLP, '98.
Tokuda;'00 - "Speech parameter generation algorithms for HMM-based speech synthesis," ICASSP, '00.
Kobayashi;'85 - "Mel generalized-log spectrum approximation...," IEICE Trans. J68-A (6), '85.
Hunt;'96 - "Unit selection in a concatenative speech synthesis system using...," ICASSP, '96.
Donovan;'95 - "Improvements in an HMM-based speech synthesiser," Eurospeech, '95.
Kawai;'04 - "XIMERA: A new TTS from ATR based on corpus-based technologies," ISCA SSW5, '04.
Hirai;'04 - "Using 5 ms segments in concatenative speech synthesis," Proc. ISCA SSW5, '04.

References (2/4)
Rouibia;'05 - "Unit selection for speech synthesis based on a new acoustic target cost," Interspeech, '05.
Huang;'96 - "Whistler: A trainable text-to-speech system," ICSLP, '96.
Mizutani;'02 - "Concatenative speech synthesis based on HMM," ASJ autumn meeting, '02.
Ling;'07 - "The USTC and iFlytek speech synthesis systems...," Blizzard Challenge workshop, 07.
Ling;'08 - "Minimum unit selection error training for HMM-based unit selection...," ICASSP, 08.
Plumpe;'98 - "HMM-based smoothing for concatenative speech synthesis," ICSLP, '98.
Wouters;'00 - "Unit fusion for concatenative speech synthesis," ICSLP, '00.
Okubo;'06 - "Hybrid voice conversion of unit selection and generation...," IEICE Trans. E89-D(11), '06.
Aylett;'08 - "Combining statistical parametric speech synthesis and unit selection..." LangTech, '08.
Pollet;'08 - "Synthesis by generation and concatenation of multiform segments," Interspeech, '08.
Yamagishi;'06 - "Average-voice-based speech synthesis," PhD thesis, Tokyo Inst. of Tech., '06.
Yoshimura;'97 - "Speaker interpolation in HMM-based speech synthesis system," Eurospeech, '97.
Tachibana;'05 - "Speech synthesis with various emotional expressions...," IEICE Trans. E88-D(11), '05.
Kuhn;'00 - "Rapid speaker adaptation in eigenvoice space," IEEE Trans. SAP 8(6), '00.
Shichiri;'02 - "Eigenvoices for HMM-based speech synthesis," ICSLP, '02.
Fujinaga;'01 - "Multiple-regression hidden Markov model," ICASSP, '01.
Nose;'07 - "A style control technique for HMM-based expressive speech...," IEICE Trans. E90-D(9), '07.
Yoshimura;'01 - "Mixed excitation for HMM-based speech synthesis," Eurospeech, '01.
Kawahara;'97 - "Restructuring speech representations using a ...", Speech Communication, 27(3), '97.
Zen;'07 - "Details of the Nitech HMM-based speech synthesis system...", IEICE Trans. E90-D(1), '07.
Abdl-Hamid;'06 - "Improving Arabic HMM-based speech synthesis quality," Interspeech, '06.

References (3/4)
Hemptinne;'06 - "Integration of the harmonic plus noise model into the...," Master thesis, IDIAP, '06.
Banos;'08 - "Flexible harmonic/stochastic modeling...," V. Jornadas en Tecnologias del Habla, '08.
Cabral;'07 - "Towards an improved modeling of the glottal source in...," ISCA SSW6, '07.
Maia;'07 - "An excitation model for HMM-based speech synthesis based on ...," ISCA SSW6, '07.
Ratio;'08 - "HMM-based Finnish text-to-speech system utilizing glottal inverse filtering," Interspeech, '08.
Drugman;'09 - "Using a pitch-synchronous residual codebook for hybrid HMM/frame...", ICASSP, '09.
Dines;'01 - "Trainable speech synthesis with trended hidden Markov models," ICASSP, '01.
Sun;'09 - "Polynomial segment model based statistical parametric speech synthesis...," ICASSP, '09.
Bulyko;'02 - "Robust splicing costs and efficient search with BMM models for...," ICASSP, '02.
Shannon;'09 - "Autoregressive HMMs for speech synthesis," Interspeech, '09.
Zen;'06 - "Reformulating the HMM as a trajectory model...", Computer Speech & Language, 21(1), '06.
Wu;'06 - "Minimum generation error training for HMM-based speech synthesis," ICASSP, '06.
Hashimoto;'09 - "A Bayesian approach to HMM-based speech synthesis," ICASSP, '09.
Wu;'08 - "Minimum generation error training with log spectral distortion for...," Interspeech, '08.
Toda;'08 - "Statistical approach to vocal tract transfer function estimation based on...," ICASSP, '08.
Oura;'08 - "Simultaneous acoustic, prosodic, and phrasing model training for TTS...," ISCSLP, '08.
Ferguson;'80 - "Variable duration models...," Symposium on the application of HMM to text speech, '80.
Levinson;'86 - "Continuously variable duration hidden...," Computer Speech & Language, 1(1), '86.
Beal;'03 - "Variational algorithms for approximate Bayesian inference," PhD thesis, Univ. of London, '03.
Masuko;'03 - "A study on conditional parameter generation from HMM...," Autumn meeting of ASJ, '03.
Yu;'07 - "A novel HMM-based TTS system using both continuous HMMs and discrete...," ICASSP, '07.

References (4/4)
Qian;'08 - "Generating natural F0 trajectory with additive trees," Interspeech, '08.
Latorre;'08 - "Multilevel parametric-base F0 model for speech synthesis," Interspeech, '08.
Tiomkin;'08 - "Statistical text-to-speech synthesis with improved dynamics," Interspeech, '08.
Toda;'07 - "A speech parameter generation algorithm considering global...," IEICE Trans. E90-D(5), '07.
Wu;'08 - "Minimum generation error criterion considering global/local variance...," ICASSP, '08.
Toda;'09 - "Trajectory training considering global variance for HMM-based speech...," ICASSP, '09.
Saino;'06 - "An HMM-based singing voice synthesis system," Interspeech, '06.
Tsuzuki;'04 - "Constructing emotional speech synthesizers with limited speech...," Interspeech, '04.
Sako;'00 - "HMM-based text-to-audio-visual speech synthesis," ICSLP, '00.
Tamura;'98 - "Text-to-audio-visual speech synthesis based on parameter generation...," ICASSP, '98.
Haoka;'02 - "HMM-based synthesis of hand-gesture animation," IEICE Technical report,102(517), '02.
Niwase;'05 - "Human walking motion synthesis with desired pace and...," IEICE Trans. E88-D(11), '05.
Hofer;'07 - "Speech driven head motion synthesis based on a trajectory model," SIGGRAPH, '07.
Ma;'07 - "A MSD-HMM approach to pen trajectory modeling for online handwriting...," ICDAR, '07.
Morioka;'04 - "Miniaturization of HMM-based speech synthesis," Autumn meeting of ASJ, '04.
Kim;'06 - "HMM-Based Korean speech synthesis system for...," IEEE Trans. Consumer Elec., 52(4), '06.
Klatt;'82 - "The Klatt-Talk text-to-speech system," ICASSP, '82.

Acknowledgement

Keiichi Tokuda would like to thank HTS working
group members, including Heiga Zen, Keiichiro
Oura, Junichi Yamagichi, Tomoki Toda, YoshihikoNankaku, Kei Hahimoto, and Sayaka Shiota for
their help.

53

HTS Slides
released by HTS Working Group
http://hts.sp.nitech.ac.jp/
Copyright (c) 1999 - 2011
Nagoya Institute of Technology
Department of Computer Science
Some rights reserved.
This work is licensed under the Creative Commons Attribution 3.0 license.
See http://creativecommons.org/ for details.

Copyright for some samples
Some samples (icon is) are released under following license.

55