Fundamentals and recent advancesin HMM-based speech synthesis

Keiichi TOKUDA

Heiga ZEN

Nagoya Institute of Technology

Toshiba Research Europe

Sept. 6, 2009

Time-line
14:15 ∼ 15:45: First half
- Fundamentals- Q&A (10min)
15:45 ∼ 16:15: Break

16:15 ∼ 17:45: Second half
- Related topics- Recent advances- Applications- Q&A (10min)

Time-line
14:15 ∼ 15:45: First half
- Fundamentals* Probabilistic formulation* HTS framework· Feature extraction· HMM· Parameter generation· Waveform reconstruction- Q&A (10min)

Speech production mechanism
F. Itakura, "Fundamentals of speech analysis andsynthesis and its application to speech coding,"
IEICE FM06-2-1, July 2006.

modulation of carrier wave
by speech information

freq. trans. char.

voiced/unvoiced

fundamental freq.

text(concept)

speechfrequencytransfercharacteristics
magnitude
start--end
fundamental
frequency

sound source
voiced: pulse
unvoiced: noise
air flowSpeech synthesis methods· Rule-based, formant synthesis (∼'90s)- Hand-crafting each phonetic units by rules- Based on source-filter model* DECtalk [Klatt;'82]· Corpus-based, concatenative synthesis ('90s∼)- Concatenate speech units (waveform) from a database- Large data + automatic learning⇒ High-quality synthetic voices can be built automatically- Single inventory: diphone synthesis [Moulines;'90]- Multiple inventory: unit selection synthesis (USS)* ATR n-Talk [Sagisaka;'92], CHATR [Black;'96]* AT&T Next-Gen TTS [Beutnagel;'99]

Speech synthesis methods· Corpus-based, statistical parametric synthesis- Proposed in mid-'90s, becomes popular since mid-'00s- Large data + automatic training⇒ Automatic voice building- Source-filter model + statistical acoustic model⇒ Flexible to change its voice characteristics- HMM as its statistical acoustic model⇒ HMM-based speech synthesis (HTS) [Yoshimura;'99]

This tutorial focuses on HMM-based speech synthesis

Time-line
14:15 ∼ 15:45: First half
- Fundamentals* Probabilistic formulation* HTS framework· Feature extraction· HMM· Parameter generation· Waveform reconstruction- Q&A (10min)

Formulation of corpus-based synthesis· Speech synthesis based on the ML criterion- Training

λ̂ = arg max p(O | W, λ)λλ : model parameters
O : training data
W : transcriptions- Synthesisô = arg max p(o | w, λ̂)oo : synthesized speechw : input text

· Using HMM as its acoustic model⇒ HMM-based speech synthesis (HTS) [Yoshimura;'02]

HMM-based speech synthesis system (HTS)· HTS: Toolkit for HMM-based speech synthesis- Web: http://hts.sp.nitech.ac.jp/- Research platform for HMM-based speech synthesis- Released as a patch code for HTK- Latest verion: HTS-2.1 (July 2008)- Speaker dependent (SD) / adaptation (SA) demo scripts* SD: HTS-demo_CMU-ARCTIC-SLT* SA: HTS-demo_CMU-ARCTIC-ADAPT

We will explain the HTS framework in details based on theimplementation of the SD demo script (HTS-2.0.1)

HMM-based speech synthesis system (HTS)SPEECH Speech signal
DATABASE
Excitation

parameter
extraction
Excitation
parameters

TEXT

Text analysis
Excitation
parameters

Synthesis part

Spectralparameter
extraction
Spectralparameters

Training HMMs

LabelsLabelsTraining partContext-dependent HMMs
& state duration models
Parameter generationfrom HMMs
Spectralparameters

Excitation Excitation Synthesis
generation
FilterSYNTHESIZEDSPEECHHMM-based speech synthesis system (HTS)
Training part

SPEECH Speech signal
DATABASE
Excitation

parameter
extraction
Excitation
parameters

Spectralparameter
extraction
Spectralparametersλ̂ = arg max p(O | W, λ)Training HMMs

LabelsTEXT

Text analysis
LabelsContext-dependent HMMs
& state duration models
Parameter generationfrom HMMsô = arg max p(o | Spectralw,
λ̂)parameters

Excitation
parameters

Synthesis part

λoExcitation Excitation Synthesis
generation
FilterSYNTHESIZEDSPEECHHMM-based speech synthesis system (HTS)SPEECH Speech signal
DATABASE
Excitation

parameter
extraction
Excitation
parameters

TEXT

Text analysis
Excitation
parameters

Synthesis part

Spectralparameter
extraction
Spectralparameters

Training HMMs

LabelsLabelsTraining partContext-dependent HMMs
& state duration models
Parameter generationfrom HMMs
Spectralparameters

Excitation Excitation Synthesis
generation
FilterSYNTHESIZEDSPEECHHMM-based speech synthesis system (HTS)SPEECH Speech signal
DATABASE
Excitation

parameter
extraction
Excitation
parameters

LabelsTEXT

Text analysis
LabelsSynthesis partTraining part
Spectralparameter
extraction
Spectralparameters

Training HMMs

Context-dependent HMMs
& state duration models
Parameter generationfrom HMMs
Excitation Excitation Synthesis
generation
FilterSYNTHESIZEDSPEECHSpeech production mechanism
F. Itakura, "Fundamentals of speech analysis andsynthesis and its application to speech coding,"
IEICE FM06-2-1, July 2006.

modulation of carrier wave
by speech information

freq. trans. char.

voiced/unvoiced

fundamental freq.

text(concept)

speechfrequencytransfercharacteristics
magnitude
start--end
fundamental
frequency

sound source
voiced: pulse
unvoiced: noise
air flowSource-filter model
Source excitation part

Vocal tract resonance part

pulse traine(n)white noise

excitation

lineartime-invariant
systemh(n)speechx(n) = h(n) ∗ e(n)x(n) = h(n) ∗ e(n)↓ Fourier transformX(ejω ) = H (ejω )E(ejω )H (ejω ) should be defined by the state-output vector of HMMs,

e.g., mel-cepstral coefficients, LSP coefficients

Maximum-likelihood estimation of spectral modelParametric models of speech spectrum
Autoregressive (AR) model,()
MX
H(z) = K
1−c(m)z −mm=0Exponential (EX) modelMX
c (m)z −m
H(z) = exp
m=0

Estimate parameters of spectral model to maximize its likelihoodc = arg max p(x | c )cp(x | c) : AR model ⇒ Linear prediction (LP) [Itakura;'70]p(x | c) : EX model ⇒ ML-based cepstral analysis

Generalized cepstral analysis [Kobayashi;'84]
Generalized cepstral coefficients: cγ (m)H(z) = s−1
γ

=








M
X

cγ (m)z −m

m=0

1+γ

M
X

m=0

!

cγ (m)z −m

M

X


−m

exp
c
(m)z
,

γ
m=0

!1/γ

, 0 < |γ| ≤ 1
γ=0Introduction of auditory frequency scalezα−1 = Ψ(z) =

−1

z −α
1 − αz −1

Phase chacteristics can be used
for frequency transformation:ω̃ = tan−1(1 − α ) sin ω(1 + α2 ) cos ω − 2α2where Ψ(ejω ) = e−j ω̃πWarped frequency ω̃ (rad)First-order all-pass function:

mel scale

π/210kHz sampling
α = 0.350π/2Frequency ω (rad)πMel-generalized cepstral analysis [Tokuda;'94]
Mel-generalized cepstral coefficients: cα,γ (m)H(z) =








1+γ

M
X

m=0

−m
cα,γ (m)zα

M

X


−m

exp
c
(m)z

α,γ
α ,
m=0

−1
z
−α
−1
zα =
1 − αz −1!1/γ, 0 < |γ| ≤ 1
γ=0

Mel-generalized cepstral analysis(α, γ) = (0, 0)⇒ Cepstral analysis(α, γ) = (0, −1)⇒ LP analysis(α, γ) = (0.42 , 0)

⇒ Mel-cepstral analysis(α, γ) = (0.42 , −1)⇒ Warped LP analysis

H(z) = exp

MXcα,γ (m)z −mm=0

H(z) =

1−

H(z) = exp

PM1

m=0 cα,γ (m)z

M
X

−m

cα,γ (m)zα−mm=0H(z) =

1−

PM

1

−m
c
(m)z
α
α,γ
m=0

A unified approach to speech spectral estimation
α=0|α| < 1, − 1 ≤ γ < 0Generalized
cepstral analysis
Linear prediction

Mel-generalized
cepstral analysis
γ = −1

Warped linear prediction
γ=0ML-based
cepstral analysis

Mel-cepstral analysis

Mel-generalized analysis of natural speech
α = 0.35

α =0

α = 0.4740

γ =0

Log Magnitude (dB)

0
-40
40

γ = - 1/2

0
-40
40

γ =-1

0
-40

0

2.5

5 0

2.5

Frequency (kHz)

5 0

2.5

5

M=16

Quantization of spectral parameters· Problems- Keep synthesis filter stable after quantization- Quantize parameters as little distortion as possible· γ = 0 (cepstral model)- Stability is always ensured for any cα,γ (m)· -1 ≤ γ < 0- Direct quantization of cα,γ (m) may cause unstable
synthesis filter- Not easy to check the filter stability⇒ Transform cα,γ (m) into other representations

Stability theorem· Transformation should be one-to-one mapping· Decomposition of polynomial [Itakura;'75]
H(z) =1+γMXcα,γ (m)zα−mm=0

C(zα ) = Cp (zα ) + Cq (zα )symmetric

!1/γ

=

1
{C (zα )}n(n = -1/γ)

antisymmetric

For all of the roots of C(zα ) to be inside the unit circle:- Zeros of Cp (zα ) & Cq (zα ) are located on the unit circle- They are simple, & they separete each otherMGC-based line spectrum pairs (MGC-LSPs)
C1 (zα ) = Cp (zα ) + Cq (zα )Cp (zα ) = C1 (zα ) + zα(M+1) C1 (zα−1 )Cq (zα ) = C1 (zα ) − zα(M+1) C1 (zα−1 )Root location of MGC-LSP [ωα (1), . . . , ωα (M )]Imizα -planeroots of Cp (zα )roots of Cq (zα )ωα (M )ωα (1)-10

1 Re

The choise of α & γ for speech analysis
Analysis/synthesis speech with fixed α & γ· Speech quality changes with γ
γ → -1 (all pole) Clearγ → 0 (cepstral) Smooth· 18-order LSP analysis gives almost the same quality as
24-order mel-cepstral analysis [Kim;'06]· When the analysis order is high enough, the difference
becomes small

Speech signal processing toolkit (SPTK)· SPTK: Toolkit for speech signal processing- Web: http://sptk.sourceforge.net/- Open source, BSD license- Supports various basic operations* Speech analysis & synthesis* Speech parameter conversion* Filtering, transforms, VQ* Data manipulation, data conversion

HTS-demo scripts adopt SPTK for speech analysis

Speech signal processing toolkit (SPTK)
AnalysisLP analysis: lpc
ML-based cepstral analysis: uels
Generalized cepstral analysis: gcep
Mel-cepstral analysis: mcep
Mel-generalized cepstral analysis: mgcep
LPC ⇔ LSP conversion: lpc2lsp, lsp2lpc
F0 detection: pitch

HTS-demo uses Snack get_f0 because of its better accuracy

HMM-based speech synthesis system (HTS)SPEECH Speech signal
DATABASE
Excitation

parameter
extraction
Excitation
parameters

TEXT

Text analysis
Excitation
parameters

Synthesis part

Spectralparameter
extraction
Spectralparameters

Training HMMs

LabelsLabelsTraining partContext-dependent HMMs
& state duration models
Parameter generationfrom HMMs
Spectralparameters

Excitation Excitation Synthesis
generation
FilterSYNTHESIZEDSPEECHHMM-based speech synthesis system (HTS)
Training part

SPEECH Speech signal
DATABASE
Excitation

Spectralparameter
extraction
Spectralparametersparameter
extraction
Excitation
parameters

Training HMMs

LabelsTEXT

Text analysis
LabelsContext-dependent HMMs
& state duration models
Parameter generationfrom HMMs

Spectralλ̂ = arg max p(Ô | W,
λ)parameters

Excitation
parameters

Synthesis part

λExcitation Excitation Synthesis
generation
FilterSYNTHESIZEDSPEECHStructure of state-output (observation) vector
otSpectrum part

Excitation partct

Mel-cepstral coefficients

D ct

D Mel-cepstral coefficients

D2c t

DD Mel-cepstral coefficientspt

log F0δpt

D log F0

δ 2 ptDD log F0

Dynamic features∂ct
D ct =≈ 0.5(ct+1 − ct−1 )
∂t
2
∂
ct
2
D ct =≈ ct+1 − 2ct + ct−1
2
∂t

ct−1

ct

ct+1

ct−1

ct

ct+1

Dct−1

Dct

Dct+1

D2 ct−1

D2 ct

D2 ct+1

Hidden Markov model (HMM)
a11
π1

1

a22
a12

b1 (ot )Observation sequence
State sequence

O o1 o2 o3 o4 o5Q2b2 (ot )

a33
a23

3
b3 (ot )... . . ... oT

1 1 1 1 2 ...

2 3 ...

3

Multi-stream HMM structureot
bj (ot )Spectrumb2j (o2t )b3j (o3t )b4j (o4t )

4

o1t

3

b1j (o1t )D2c t

Excitation

s=1

D ct

Stream 1 2SY¡ s s ¢ws=
bj (ot )bj (ot )

ct

pt
δ pt
δ 2 pt

o2t
o3t
o4t

Log Frequency

Observation of F0

Time

Unable to model by continuous or discrete distribution

MSD-HMM for F0 modeling1

HMM for F0Ω1 = R0Ω2 = R1

w1,0.

unvoiced

w1,1
voicedvoiced/unvoiced weights

2

w2,0

.

unvoiced

w2,1
voiced3

w3,0.

unvoiced

w3,1
voicedStream 1

Structure of state-output distributions

Mel-cepstrum
VoicedStream 2, 3, 4

Unvoicedlog F0

VoicedUnvoiced
VoicedUnvoiced

Single Gaussian

MSD(Gaussian & discrete)
MSD(Gaussian & discrete)
MSD(Gaussian & discrete)

Stream 1

Proto-type definition (from HTS-demo)

MCEP
Voicedlog F0Stream 2, 3, 4

Unvoiced
VoicedUnvoiced
VoicedUnvoiced~o <VecSize> 78 <USER> <DIAGC><StreamInfo> 4 75 1 1 1 <MSDInfo> 4 0 1 1 1<BeginHMM> <NumStates> 7 <State> 2<Stream> 1<Mean> 75
0.0 0.0 0.0 0.0 0.0 ...
<Variance> 75
1.0 1.0 1.0 1.0 1.0 ...
<Stream> 2<NumMixes> 2<Mixture> 1 0.5000<Mean> 1
0.0<Variance> 1
1.0<Mixture> 2 0.5000<Mean> 0<Variance> 0<Stream> 3 ...

Training process (from HTS-demo)
data & labels

Compute variancefloor (HCompV)

Reestimate CD-HMMs by
EM algorithm (HERest)

Estimate CD-dur. models
from FB stats (HERest)

Initialize CI-HMMs bysegmental k-means (HInit)

Decision tree-based
clustering (HHEd TB)

Decision tree-based
clustering (HHEd TB)

Reestimate CI-HMMs by
EM algorithm(HRest & HERest)

Reestimate CD-HMMs
by EM algorithm (HERest)

Copy CI-HMMs to
CD-HMMs (HHEd CL)

Untie parameter tying
structure (HHEd UT)monophone(context-independent, CI)

fullcontext(context-dependent, CD)

Estimated dur models
Estimated
HMMs

Training process (from HTS-demo)
data & labels

Compute variancefloor (HCompV)

Reestimate CD-HMMs by
EM algorithm (HERest)

Estimate CD-dur. models
from FB stats (HERest)

Initialize CI-HMMs bysegmental k-means (HInit)

Decision tree-based
clustering (HHEd TB)

Decision tree-based
clustering (HHEd TB)

Reestimate CI-HMMs by
EM algorithm(HRest & HERest)

Reestimate CD-HMMs
by EM algorithm (HERest)

Copy CI-HMMs to
CD-HMMs (HHEd CL)

Untie parameter tying
structure (HHEd UT)monophone(context-independent, CI)

fullcontext(context-dependent, CD)

Estimated dur models
Estimated
HMMs

Training process (from HTS-demo)
data & labels

Compute variancefloor (HCompV)

Reestimate CD-HMMs by
EM algorithm (HERest)

Estimate CD-dur. models
from FB stats (HERest)

Initialize CI-HMMs bysegmental k-means (HInit)

Decision tree-based
clustering (HHEd TB)

Decision tree-based
clustering (HHEd TB)

Reestimate CI-HMMs by
EM algorithm(HRest & HERest)

Reestimate CD-HMMs
by EM algorithm (HERest)

Copy CI-HMMs to
CD-HMMs (HHEd CL)

Untie parameter tying
structure (HHEd UT)monophone(context-independent, CI)

fullcontext(context-dependent, CD)

Estimated dur models
Estimated
HMMs

Training process (from HTS-demo)
data & labels

Compute variancefloor (HCompV)

Reestimate CD-HMMs by
EM algorithm (HERest)

Estimate CD-dur. models
from FB stats (HERest)

Initialize CI-HMMs bysegmental k-means (HInit)

Decision tree-based
clustering (HHEd TB)

Decision tree-based
clustering (HHEd TB)

Reestimate CI-HMMs by
EM algorithm(HRest & HERest)

Reestimate CD-HMMs
by EM algorithm (HERest)

Copy CI-HMMs to
CD-HMMs (HHEd CL)

Untie parameter tying
structure (HHEd UT)monophone(context-independent, CI)

fullcontext(context-dependent, CD)

Estimated dur models
Estimated
HMMs

Training process (from HTS-demo)
data & labels

Compute variancefloor (HCompV)

Reestimate CD-HMMs by
EM algorithm (HERest)

Estimate CD-dur. models
from FB stats (HERest)

Initialize CI-HMMs bysegmental k-means (HInit)

Decision tree-based
clustering (HHEd TB)

Decision tree-based
clustering (HHEd TB)

Reestimate CI-HMMs by
EM algorithm(HRest & HERest)

Reestimate CD-HMMs
by EM algorithm (HERest)

Copy CI-HMMs to
CD-HMMs (HHEd CL)

Untie parameter tying
structure (HHEd UT)monophone(context-independent, CI)

fullcontext(context-dependent, CD)

Estimated dur models
Estimated
HMMs

Context-dependent modeling· Current phoneme· {preceding, succeeding} two phonemes· Position of current phoneme in current syllable· # of phonemes at {preceding, current, succeeding} syllable· {accent, stress} of {preceding, current, succeeding} syllable· Position of current syllable in current word
· # of {preceding, succeeding} {stressed, accented} syllables in current phrase· # of syllables {from previous, to next} {stressed, accented} syllable· Vowel within current syllable· Guess at part of speech of {preceding, current, succeeding} word
· # of syllables in {preceding, current, succeeding} word
· Position of current word in current phrase· # of {preceding, succeeding} content words in current phrase· # of words {from previous, to next} content word
· # of syllables in {preceding, current, succeeding} phrase...

Vast # of combinations ⇒ Difficult to have all possible models

Training process (from HTS-demo)
data & labels

Compute variancefloor (HCompV)

Reestimate CD-HMMs by
EM algorithm (HERest)

Estimate CD-dur. models
from FB stats (HERest)

Initialize CI-HMMs bysegmental k-means (HInit)

Decision tree-based
clustering (HHEd TB)

Decision tree-based
clustering (HHEd TB)

Reestimate CI-HMMs by
EM algorithm(HRest & HERest)

Reestimate CD-HMMs
by EM algorithm (HERest)

Copy CI-HMMs to
CD-HMMs (HHEd CL)

Untie parameter tying
structure (HHEd UT)monophone(context-independent, CI)

fullcontext(context-dependent, CD)

Estimated dur models
Estimated
HMMs

Training process (from HTS-demo)
data & labels

Compute variancefloor (HCompV)

Reestimate CD-HMMs by
EM algorithm (HERest)

Estimate CD-dur. models
from FB stats (HERest)

Initialize CI-HMMs bysegmental k-means (HInit)

Decision tree-based
clustering (HHEd TB)

Decision tree-based
clustering (HHEd TB)

Reestimate CI-HMMs by
EM algorithm(HRest & HERest)

Reestimate CD-HMMs
by EM algorithm (HERest)

Copy CI-HMMs to
CD-HMMs (HHEd CL)

Untie parameter tying
structure (HHEd UT)monophone(context-independent, CI)

fullcontext(context-dependent, CD)

Estimated dur models
Estimated
HMMs

Decision tree-based state clustering [Odell;'95]
k-a+b
t-a+n
L=voice?

R=silence?
yes

L="w" ?
yes

yes

no

no

yes

no

R=silence?
noyes

L="gy" ?
noleaf nodes

synthesizedstates

w-a+t

w-a+silgy-a+sil

w-a+shg-a+silgy-a+pau

Stream-dependent tree-based clustering
Spectrum & excitation have different context dependency⇒ Build decision trees separately

Decision treesformel-cepstrum
Decision trees
for F0Training process (from HTS-demo)
data & labels

Compute variancefloor (HCompV)

Reestimate CD-HMMs by
EM algorithm (HERest)

Estimate CD-dur. models
from FB stats (HERest)

Initialize CI-HMMs bysegmental k-means (HInit)

Decision tree-based
clustering (HHEd TB)

Decision tree-based
clustering (HHEd TB)

Reestimate CI-HMMs by
EM algorithm(HRest & HERest)

Reestimate CD-HMMs
by EM algorithm (HERest)

Copy CI-HMMs to
CD-HMMs (HHEd CL)

Untie parameter tying
structure (HHEd UT)monophone(context-independent, CI)

fullcontext(context-dependent, CD)

Estimated dur models
Estimated
HMMs

Training process (from HTS-demo)
data & labels

Compute variancefloor (HCompV)

Reestimate CD-HMMs by
EM algorithm (HERest)

Estimate CD-dur. models
from FB stats (HERest)

Initialize CI-HMMs bysegmental k-means (HInit)

Decision tree-based
clustering (HHEd TB)

Decision tree-based
clustering (HHEd TB)

Reestimate CI-HMMs by
EM algorithm(HRest & HERest)

Reestimate CD-HMMs
by EM algorithm (HERest)

Copy CI-HMMs to
CD-HMMs (HHEd CL)

Untie parameter tying
structure (HHEd UT)monophone(context-independent, CI)

fullcontext(context-dependent, CD)

Estimated dur models
Estimated
HMMs

Estimation of state duration models [Yoshimura;'98]
t1

i

t0

1

2

χt0 ,t1 (i) ∝X

αt0 −1(j)aji · atii1 −t0 ··aik bk (ot1 +1)βt1 +1(k)

j6=i

3

X
k6=i

4

5

6

7
t1
Y

T=8

bi (ot )t=t 0tStream-dependent tree-based clusteringState duration
model
HMM
Decision treesformel-cepstrum
Decision trees
for F0Decision tree
for state dur.
modelsTraining process (from HTS-demo)
data & labels

Compute variancefloor (HCompV)

Reestimate CD-HMMs by
EM algorithm (HERest)

Estimate CD-dur. models
from FB stats (HERest)

Initialize CI-HMMs bysegmental k-means (HInit)

Decision tree-based
clustering (HHEd TB)

Decision tree-based
clustering (HHEd TB)

Reestimate CI-HMMs by
EM algorithm(HRest & HERest)

Reestimate CD-HMMs
by EM algorithm (HERest)

Copy CI-HMMs to
CD-HMMs (HHEd CL)

Untie parameter tying
structure (HHEd UT)monophone(context-independent, CI)

fullcontext(context-dependent, CD)

Estimated dur models
Estimated
HMMs

HMM-based speech synthesis system (HTS)SPEECH Speech signal
DATABASE
Excitation

parameter
extraction
Excitation
parameters

TEXT

Text analysis
Excitation
parameters

Synthesis part

Spectralparameter
extraction
Spectralparameters

Training HMMs

LabelsLabelsTraining partContext-dependent HMMs
& state duration models
Parameter generationfrom HMMs
Spectralparameters

Excitation Excitation Synthesis
generation
FilterSYNTHESIZEDSPEECHHMM-based speech synthesis system (HTS)
Training part

SPEECH Speech signal
DATABASE
Excitation

parameter
extraction
Excitation
parameters

Spectralparameter
extraction
Spectralparametersô = arg max p(o | w, λ̂)
Training HMMs

LabelsTEXT

Text analysis
LabelsExcitation
parameters

Synthesis partoContext-dependent HMMs
& state duration models
Parameter generationfrom HMMs
Spectralparameters

Excitation Excitation Synthesis
generation
FilterSYNTHESIZEDSPEECHSpeech parameter generation algorithm [Tokuda;'00]ô = arg max p(o | w, λ̂)o= arg maxoX∀qp(o, q | w, λ̂)≈ arg max max p(o | q, λ̂) p(q | w, λ̂)oq⇓q̂ = arg max P (q | w, λ̂)qô = arg max p(o | q̂, λ̂)oSpeech parameter generation algorithm [Tokuda;'00]ô = arg max p(o | w, λ̂)o= arg maxoX∀qp(o, q | w, λ̂)≈ arg max max p(o | q, λ̂) p(q | w, λ̂)oq⇓q̂ = arg max P (q | w, λ̂)qô = arg max p(o | q̂, λ̂)oDetermination of state sequence (1)
a11
π1

1

a22
a12

b1 (ot )Observation sequence

O o1 o2 o3 o4 o5State sequence

Q

State duration

D2b2 (ot )a233b3 (ot )... . . ... oT

1 1 1 1 2 ...
4

a33

10

2 3 ...

3

5

Determine state sequence via determining state durations

Determination of state sequence (2)P (q | w, λ̂) =KYpi (di )i=1

pi (·) : state-duration distribution of i-th state
di : state duration of i-th state
K : # of states in a sentence HMM for w

Determination of state sequence (3)Geometricdi −1
pi (di ) = aii (1 − aii )⇒ dˆi = 1

Gaussianˆd =m2
pi (di ) = N (di ; mi , σi ) ⇒iiState-duration prob.

0.5
0.4
0.3
0.2
0.1
0.0

1

2

3

4

5

6

7

8

State duration

Speech parameter generation algorithm [Tokuda;'00]ô = arg max p(o | w, λ̂)o= arg maxoX∀qp(o, q | w, λ̂)≈ arg max max p(o | q, λ̂) p(q | w, λ̂)oq⇓q̂ = arg max P (q | w, λ̂)qô = arg max p(o | q̂, λ̂)oWithout dynamic features

Mean

Variance⇓oˆ becomes a sequence of mean vectors

Integration of dynamic features
Speech param. vec. ot includes both static & dyn. feats.

£ >
¤
> >
ot = ct , D ct
MM

D ct = ct − ct−1
c t-2

c t-1

ct

c t+1

c t+2

Dct-2

Dc t-1

Dc t

Dc t+1

Dct+2

2M

The relationship between ot & ct can be arranged as


o
..
.





 ct−1 ot−1D ct−1  ct o t
 D ct  ct+1 ot+1D ct+1 


..
.



· · ·
· · ·

· · ·

· · ·= 
· · ·

· · ·

· · ·

···

..
.
0
−I
0
0
0
0
..
.

W

..
.
I
I
0
−I
0
0
..
.

..
.
0
0
I
I
0
−I
..
.

..
.
0
0
0
0
I
I
..
.


· · ·
· · ·
· · ·
· · ·
· · ·
· · ·
· · ·

···

c



..
 . 
ct−2 
ct−1  ct 
ct+1 


..
.

W

1 0 0 ...
1 -1 0 ...

...

... 0 1 -1... 0 1 0... 0 0 0

1 0 0 ...
1 -1 0 ...

0 1 0 ...
0 1 -1 ...

>

0

>

Σ−1
q̂
0

1 0 0 ...
0 0 0 ...

c1
c2

0 1 0 ...
-1 1 0 ...

...

W

0 1 0 ...
0 1 -1 ...

...

... 0 1 -1... 0 1 0... 0 0 0... 0 0 1...

=... 0 0 1

Solution for the problem
−1
Σq̂

W

... 0 1 00... -1 1 0... 0 0 1... 0 -1 1

µq̂
µq1
µq2

0
µqT

c

cT

Dynamic

StaticGenerated speech parameter trajectory

Mean

VariancecGenerated spectraSynthesis tools in HTS· HMGenS- Tool to generate speech parameters from HMMs
- Requires HTK/HTS library modules- Many functionalities⇒ For research purpose· hts_engine- Web: http://hts-engine.sourceforge.net/- Small run-time synthesis engine including waveform synth.
- API & stand-alone program- Works without HTK/HTS library⇒ For application development purpose

HTS-demo demonstrates how to use both tools

HMM-based speech synthesis system (HTS)SPEECH Speech signal
DATABASE
Excitation

parameter
extraction
Excitation
parameters

TEXT

Text analysis
Excitation
parameters

Synthesis part

Spectralparameter
extraction
Spectralparameters

Training HMMs

LabelsLabelsTraining partContext-dependent HMMs
& state duration models
Parameter generationfrom HMMs
Spectralparameters

Excitation Excitation Synthesis
generation
FilterSYNTHESIZEDSPEECHHMM-based speech synthesis system (HTS)SPEECH Speech signal
DATABASE
Excitation

parameter
extraction
Excitation
parameters

TEXT

Text analysis
Excitation
parameters

Synthesis part

Spectralparameter
extraction
Spectralparameters

Training HMMs

LabelsLabelsTraining partContext-dependent HMMs
& state duration models
Parameter generationfrom HMMs
Spectralparameters

Excitation Excitation Synthesis
generation
FilterSYNTHESIZEDSPEECH

Source-filter model
Generated
excitation parameter(log F0 with V/UV)Generated
spectral parameter(cepstrum, LSP)

pulse traine(n)white noise

excitation

lineartime-invariant
systemh(n)synthesized
speechx(n) = h(n) ∗ e(n)Spectral representation & corresponding filter
cepstrum: LMA filter
generalized cepstrum: GLSA filter
mel-cepstrum: MLSA filter
mel-generalized cepstrum: MGLSA filter
LSP: LSP filter
PARCOR: all-pole lattice filter
LPC: all-pole filter

Structure of synthesis filter H(z) (γ = 0)
H(z) = exp

M
XExponential function⇒ digital filter

cα,γ (m)zα−m

m=0↓ Pade approximation
≈ RL

M
X

m=0

cα,γ (m)zα−m!

Rational function⇒ digital filter· Sufficient accuracy: maximum spectral error 0.24dB· O(8M) multiply-add operations per sample· Stable filter· M multiply-add operations for filter coefficients calculation

Speech signal processing toolkit (SPTK)
Synthesis
Excitation signal generation: excite
LMA filter (cepstrum): lmadf
GLSA filter (generalized cepstrum): glsadf
MLSA filter (mel-cepstrum): mlsadf
MGLSA filter (mel-generalized cepstrum): mglsadf
LSP filter (LSP): lspdf
All-pole lattice filter (PARCOR): ltcdf
All-pole filter (LPC): poledf
FIR filter: zerodf

Speech samples
Mel-cepstrum
w/ dyn.
log F0w/ dyn.
w/o dyn.

w/o dyn.

Summary (first half)
Fundamentals in HMM-based speech synthesis· Probabilistic formulation of corpus-based synthesis· HMM-based speech synthesis system- Spectral analysis- Acoustic modeling (HMM)- Parameter generation- Speech synthesis filter

Any questions?

Time-line
16:15 ∼ 17:45: Second half
- Related topics* Unit selection & hybrid* Flexibility to control voice characteristics· Adaptation, interpolation, eigenvoice, multiple regress.

- Recent advances* Vocoding* Acoustic modeling* Over-smoothing compensation- Applications- Q&A (10min)

Time-line
16:15 ∼ 17:45: Second half
- Related topics* Unit selection & hybrid* Flexibility to control voice characteristics· Adaptation, interpolation, eigenvoice, multiple regress.

- Recent advances* Vocoding* Acoustic modeling* Over-smoothing compensation- Applications- Q&A (10min)

Unit-selection synthesis (USS) (1)
SPEECH Speech signal
DATABASE
Prosodicparameter
extraction
Prosodicparameters

Transcriptions

Training part
Spectralparameter
extraction
Spectralparameters

Build USS system

Prosody
predictors

Unit database& cost functions

TEXT

Text analysis
Prosody
prediction

Synthesis partUnit-selection
algorithm
Signalprocessing

SYNTHESIZEDSPEECH

Unit-selection synthesis (USS) (2)Unit-selection algorithm [Hunt;'96]
candidateunits
targetunits

concatenationui−1 cost ui

ui+ 1

C c (ui−1 , ui ) =target
cost

ti−1

ti

n
n
ûn1 = arg min
{C(t
,
u
1
1 )}
n
u1

· Major techniques

C t (ti , ui ) =t i+ 1C(tn1 , un1 ) =

q
X

k=1
p
X

wkc Ckc (ui−1 , ui )wjt Cjt (ti , ui )

j=1

n
X

C t (ti , ui ) +i=1- General selection technique [Hunt;'96]- Clustering-based technique [Donovan;'95]

n
X
i=2

C c (ui−1 , ui )

General unit-selection synthesis scheme
All segments

Target cost

Concatenation costClustering-based unit-selection synthesis scheme

Clusteredsegments

Relation between two approaches (1)Unit selection

HMM-based synthesis

Clusteredsegments

ClusteredstatesMergedstatesDelta

StaticSentence
HMM

Gaussian

ML trajectory

Relation between two approaches (2)Unit selection

HMM-based synthesis

TEXT

TEXT

Text analysis

Text analysis

Features

Dur.

LabelsF0
Spect.

Dur.

F0

Units

Synthesis

Synthesis

SYNTHESIZEDSPEECHSYNTHESIZEDSPEECH

Unit selection: Serial

HMM-based: Parallel

Relation between two approaches (3)Approximate state-output distribution by templatesContinuousx x x x x x x xx xxx xx x x x x xxxx x xx x x x x xxxxxxx x xxx xx xxx xxxx
xx x xxx xx xxxxxxxx x xx x
x
x
x
x
x x
x
x
Acoustic space x: data⇒approxDiscretex x x x x x x xx xxx xx x x x x xxxx x xx x x x x xxxxxxx x xxx xx xxx xxxx
xx x xxx xx xxxxxxxx x xx x
x
x
x
x
x x
x
x
Acoustic space x: data

ML-based param. generation with discrete distribution⇒ Result in frame-wise DP search like unit selection

Parameter generation ⇒ "analogue" version of USHybrid approaches (1)Hybrid between unit-selection & HMM-based synth.
· Target prediction- HMM outputs as targets- HMM probabilities as costs· Smoothing units- Smooth discontinuities between units· Mixing units- Mix natural & generated unitsHybrid approaches (2)Target prediction· HMM outputs as targets- MFCCs, F0s, durations from HMMs as target* ATR [Kawai;'04], Arcadia [Hirai;'04], France Telecom [Rouibia;'05]· HMM probabilities as costs- Static features ⇒ Target costs- Dynamic features ⇒ Concatenation costs* Microsoft [Huang;'96], Nitech [Mizutani;'02], iFlyTek [Ling;'07]· Minimum unit-selection error training [Ling;'08]- Loss func.: # of diff. units between selected & natural units- Optimize HMM params. to minimize the loss func. by GPD

Hybrid approaches (3)Smoothing units
TEXT
Text analysis

Spectra &state sequence
Spectralsmoother

Unit selection& concation
Smoothed
spectraResidual
Waveform
synthesizer

SYNTHESIZEDSPEECH

Smooth unit boundaries using HMM statistics- Microsoft [Plumpe;'98], OGI [Wouters;'00]

Hybrid approaches (4)Mixing units
Natural units

Generated units

Mix natural & generated units⇒ Avoid data sparsity & produce large proportion of speech- Waseda Univ. [Okubo;'06], Cereproc [Aylett;'08], Nuance [Pollet;'08]

Hybrid approaches (5)Issues in hybrid approach· Non-parametric approach- Store templates & statistics→ Increase footprint- Generation + search→ Increase computational cost
- Speech is synthesized from templates & statistics→ Lose flexibility to change its voice characteristics

Time-line
16:15 ∼ 17:45: Second half
- Related topics* Unit selection & hybrid* Flexibility to control voice characteristics· Adaptation, interpolation, eigenvoice, multiple regress.

- Recent advances* Vocoding* Acoustic modeling* Over-smoothing compensation- Applications- Q&A (10min)Adaptation (Mimicking voices)
Adaptation/adaptive training of HMMs

- Originally developed in ASR, but works very well in TTS- Average voice-based speech synthesis (AVSS) [Yamagishi;'06]
Average-voice model

Training speakers

Adaptive
Training

Adaptation
Target speakers- Require small data of target speaker/speaking style⇒ Smal cost to create new voices

Adaptation demo
· Speaker adaptation- VIP voice: GWB- Child voice:

BHO· Style adaptation (in Japanese)- Joyful- Sad- Rough

From http://homepages.inf.ed.ac.uk/jyamagis/Demo-html/demo.html

Interpolation (Mixing voices)
Interpolate parameters among representive HMM sets

- Can obtain new voices even no adaptation data is available- Gradually change spkrs. & speaking styles [Yoshimura;'97,Tachibana;'05]
λ2
λ1
I(λ0 , λ2)I(λ0 , λ1)

λ : HMM set

I(λ0 , λ ) : Interpolation ratioλ0
I(λ0 , λ3)I(λ0 , λ4)λ4

λ3

Interpolation demo
· Speaker interpolation (in Japanese)- Male & Female

Male

Female· Style interpolation- Neutral → Angry- Neutral → Happy

From http://www.sp.nitech.ac.jp/& http://homepages.inf.ed.ac.uk/jyamagis/Demo-html/demo.html

Eigenoice (Producing voices) [Shichiri;'02]· Increase # of representative HMM sets in interpolation⇒ Difficult to set interpolation ratio to obtain desired voice· Eigenvoice [Kuhn;'00]: Apply PCA to super-vectors⇒ Can reduce the dimensionality of speaker space
Eigenvectors

s6

s4

s0
s2

e2
s5
s7
Mean vector µ̄

s3

e1

s1

Reduced space

Eigenvoice demo
Speaker characteristics modification
Weights for eigenvectors
+30

1st

2nd

3rd

4th

5th

Weights for eigenvectors
+30

0

0

-30

-30
Weights for eigenvectors

+30

1st

2nd

3rd

4th

5th

1st

2nd

3rd

4th

5th

Weights for eigenvectors
+30

0

0

-30

-30

1st

2nd

3rd

From http://www.sp.nitech.ac.jp/~demo/synthesis_demo_2001/

4th

5th

Multiple-regression (Controlling voices)· No eigen-vectors represents specific physical meaning⇒ Difficult to control voice characteristics intutively· Multiple-regression HMMs [Fujinaga;'01]- Assign intutive meaning to control synthetic voice [Nose;'07]
Angry

Emotion vector>z = [za , zh , zs ]Sad

Emotion spaceHappyµj = Mj ξ,£ξ = 1, z¤> >µj : mean vector of j-th state
Mj : regression matrix
ξ : extended control vector

Multiple-regression demo
Style-control
Rough

Sad

JoyfulFrom http://homepages.inf.ed.ac.uk/jyamagis/Demo-html/demo.html

Time-line
16:15 ∼ 17:45: Second half
- Related topics* Unit selection & hybrid* Flexibility to control voice characteristics· Adaptation, interpolation, eigenvoice, multiple regress.

- Recent advances* Vocoding* Acoustic modeling* Over-smoothing compensation- Applications- Q&A (10min)

Drawbacks of HMM-based speech synthesis· The biggest drawback of HTS is quality· Three major factors- Poor vocoding* How to parameterize speech waveform?

- Inaccurate acoustic modeling* How to model extracted speech parameter trajectories?

- Over-smoothing* How to recover generated speech parameter trajectories?

Recent improvements drastically have enhanced thequality of HMM-based speech synthesis

Time-line
16:15 ∼ 17:45: Second half
- Related topics* Unit selection & hybrid* Flexibility to control voice characteristics· Adaptation, interpolation, eigenvoice, multiple regress.

- Recent advances* Vocoding* Acoustic modeling* Over-smoothing compensation- Applications- Q&A (10min)

Excitation model
pulse traine(n)white noise

excitation
Unvoiced

Voiced· Excitation model: pulse/noise excitation- voiced (periodic) → pulse trains- unvoiced (aperiodic) → white noise· Difficult to model mix of V/UV excitations (e.g., V. fricatives)· Synthesized speech sounds buzzy

Advanced vocoders & excitation models· MELP-style mixed excitation [Yoshimura;'01]· STRAIGHT [Kawahara;'99, Zen;'07]· Multi-band excitation [Abdel-Hamid;'06]· Harmonic plus noise model (HNM) [Hemptinne;'06]· Harmonic / stochastic model [Banos;'08]· Glottal-flow derivative model (LF model) [Cabral;'07]· ML excitation [Maia;'07]· Glottal waveform [Raitio;'08]· Residual codebook [Drugman;'09]

70

70

30
20
0

0

1

2
3
Frequency (kHz)

4

80
70
60
50
40
30
20

40
30
20

80

10

70

0

0

1
2
3
Frequency (kHz)

4

80
70

0

1

2
3
Frequency (kHz)

4

60

50
40
30
20
0

50
40

0

1
2
3
Frequency (kHz)

Mixed Excitation

30
20
0

60

10

10

10
0

50

Log magnitude (dB)4060

⇓ Mix ⇓

50

Log magnitude (dB)60

Log magnitude (dB)80

⇓ Bandpass filtering ⇓Log magnitude (dB)80

10Log magnitude (dB)

Noise excitation

Pulse excitation

MELP-style mixed excitation (1)

0

1
2
3
Frequency (kHz)44

MELP-style mixed excitation (2)Amplitude

12

-12

0

1144

2288

3432

4576

5720

6864

8008

9152

10296 sample

1144

2288

3432

4576

5720

6864

8008

9152

10296 sample

k

o

sh

I

z

u

ts

u

Amplitude

12

-12

0

s

U

MELP-style mixed excitation (3)Preference score (%)

100
80
60.5

60

66.0
56.8

55.9

40
20
0

10.7
Pulse
or
Noise

Mixed
ME+
FM+
AP+
excitation Fourier
aperiodic
pulse
(ME)
magnitude
pulse
dispersion
(FM)
(AP)
(PD)

STRAIGHT
Waveform

Synthetic waveform

F0 extraction

Synthesis

Fixed-point analysis
AnalysisF0 adaptive spectral
smoothing in thetime-frequency region

F0

Mixed excitation withphase manipulation

Smoothed
spectrum
Aperiodic
factors

STRAIGHT + ML-based mel-cepstral analysis
120

FFT power spectrum
FFT + mel-cepstral analysis
STRAIGHT + mel-cepstral analysis

100

Power [dB]

80
60
40
20
0
-20

0

2

4

Frequency [kHz]

6

8

8
4
0

T

2T

0
-0.2
-0.4
0

Pulse excitation

8
4

0
-4

Noise excitation

0

8

WeightingPhase manipulation

Weighting

Mixed excitation
8
4

Aperiodicity

T

2T

Power [dB]

Amplitude

F0

2
4
6
Frequency [kHz]

-5

-15
-25
-35

0

2
4
6
Frequency [kHz]Amplitude

0-4Power [dB]

Amplitude

Excitation signal generation in STRAIGHT

8

0
-4
0

T

2T

Effect of STRAIGHT [Zen;'07]

Average MOS

5

4.7

4.7

G

H

4
3.1

3
2
1

2.0

A

3.2

3.2

3.1

D

E

F

2.0

B

C

Spectra 24 mcep. 24 ST. 24 mcep. 24 ST. 39 ST. 24 mcep. 24 ST.
Model
HMM
HMM
HSMM HSMM HSMM HSMM HSMM
Par Gen non-GV non-GV non-GV non-GV non-GV
GV
GV

39 ST.
HSMM
GV

ML excitation [Maia;'07]
Sentence
HMM
Mel-cepstralcoefficients

ct-2

ct-1

ct

c t+1

c t+2

log F0values

pt-2

pt-1

pt

p t+1

p t+2

Filters

Hv (z), Hu (z)Pulse train t(n)generator

White noisew(n)Hv (z)Hu (z)v(n)Voicedexcitationu(n)Unvoiced
excitation

e(n)
Mixed
excitation

H(z)Synthesized
speechML excitation STRAIGHT Pulse/noise

NaturalExamples of excitation signals
0
0

0

0
0
0

0
0

Upper: Waveform

Lower: excitation (residual)

Time-line
16:15 ∼ 17:45: Second half
- Related topics* Unit selection & hybrid* Flexibility to control voice characteristics· Adaptation, interpolation, eigenvoice, multiple regress.

- Recent advances* Vocoding* Acoustic modeling* Over-smoothing compensation- Applications- Q&A (10min)

Acoustic modeling research· Improve accuracy of p(O | W, λ)- Trended HMM [Dines;'01]- Polynomial segment model [Sun;'09]- Buried Markov model (BMM) [Bulyko;'02]- AR-HMM [Shannon;'09]- HSMM [Zen;'04]- Trajectory HMMs [Zen;'04]- Minimum generation error (MGE) training [Wu;'06]· Relax approximations- Bayesian approach [Hashimoto;'09]- MGE log-spectral distortion (MGE-LSD) [Wu;'08]- Statistical vocoder (STAVOCO) [Toda;'08]- Joint front-end/back-end training [Oura;'08]

Acoustic modeling research· Improve accuracy of p(O | W, λ)- Trended HMM [Dines;'01]- Polynomial segment model [Sun;'09]- Buried Markov model (BMM) [Bulyko;'02]- AR-HMM [Shannon;'09]- HSMM [Zen;'04]- Trajectory HMMs [Zen;'04]- Minimum generation error (MGE) training [Wu;'06]· Relax approximations- Bayesian approach [Hashimoto;'09]- MGE log-spectral distortion (MGE-LSD) [Wu;'08]- Statistical vocoder (STAVOCO) [Toda;'08]- Joint front-end/back-end training [Oura;'08]

Acoustic modeling
Limitations of HMMs for modeling speech· Piece-wise constant statistics- Statistics do not vary within an HMM state· Frame-wise
independence
assumption/sil/conditional
/a/
/i/
/sil/
- State output prob. depends only on the current state· Weak duration modeling- State2.2duration prob. decreases exponentially with timec(2)0None of themhold for real speech
-1.2
0.4

D c(2)

0
Speech params.
are generated from acoustic models-0.5⇒ Bettermean
acoustic model may produce better speechvariance

Acoustic modeling
Limitations of HMMs for modeling speech· Piece-wise constant statistics- Statistics do not vary within an HMM state· Frame-wise conditional independence assumption- State output prob. depends only on the current state· Weak duration modeling- State duration
q prob. decreasesqqexponentially with time

1

2

3

o1

o2

o3

None of them hold for real speech
SPS generates speech directly from AM⇒ More precise AM may produce better speech

Acoustic modeling
Limitations of HMMs for modeling speech· Piece-wise constant statistics- Statistics do not vary within an HMM state· Frame-wise conditional independence assumption- State output prob. depends only on the current state· Weak duration modeling- State duration prob. decreases exponentially with time

None of them hold for real speechState-duration prob.

0.5
0.4

SPS generates speech directly from AM⇒ More precise AM may produce better speech
0.3
0.2
0.1
0.0

1

2

3

4

5

6

7

8

State duration

Acoustic modeling
Limitations of HMMs for modeling speech· Piece-wise constant statistics- Statistics do not vary within an HMM state· Frame-wise conditional independence assumption- State output prob. depends only on the current state· Weak duration modeling- State duration prob. decreases exponentially with time

None of them hold for real speech
Speech params. are generated from acoustic models⇒ Better acoustic model may produce better speechBetter acoustic modeling (1)Advanced acoustic model· Piece-wise constant statistics ⇒ Dynamical model- Capture dynamics of speech parameter trajectory* Trended HMM [Dines;'01]* Polynomial segment model [Sun;'09]· Conditional indep. assumption ⇒ Graphical model- Additional dependency between variables* Buried Markov model [Bulyko;'02]* Autoregressive HMM [Shannon;'09]· Weak duration modeling ⇒ Explicit duration model- State duration PDFs are explicitly modeled* Hidden semi-Markov model [Zen;'07]

Better acoustic modeling (2)Trajectory HMM [Zen;'06]· Derived from HMM with dynamic feature constraints· Underlying generative model of HMM-based synthesis

P (c | λ) =X∀qP (c | q, λ) P (q | λ)P (c | q, λ) = N (c | c̄q , Pq )−1
Rq = W > Σ−1
W
=
P
q
q

rq = W > Σ−1q µqc̄q = Pq rq

Better acoustic modeling (3)mean trajectory c̄q

sil

a

i

d

a

sil
sil

5
10

a

15

Covariance matrix Pq is full

⇒ All state output probabilities
depend on each other

Limitations of HMMs can be avoided

20
25
i

35

Statistics vary according to theirneighboring models

a

30
d

Time (frame)c̄q is given as a smooth trajectory⇒ Statistics vary within a state⇒ Coarticulation effect can naturally
be captured

40
45
50
55
sil

5

10

15

20

25
30
35
Time (frame)

40

45

50

Temporal covariance matrix Pq

55

Better acoustic modeling (4)Relation between HMM-based synth. & trajectory HMM· Mean vector of trajectory HMM,

· ML trajectory by speech param. generation algorithm, candare identicalBetter acoustic modeling (4)AM training of trajectory HMM

λ̂ = arg max {p(C | W, λ)} : ML [Zen;'06]λλ̂ = arg min {E(C; W, λ)} : MMSE (MGE) [Wu;'06]
λMinimize error betw. training & generated trajectories

Training
Synthesis

Conventionalp(O | W, λ)p(c | w, λ̂)

trajectory HMM
p(C | W, λ)p(c | w, λ̂)

Solve inconsistency between training & synthesis

Better acoustic modeling (5)Effect of trajectory HMM [Zen;'07]0Preference Score (%)

20

40

Baum-Welch

42.5%

Viterbi

38.4%

60

80

100

69.1%

trajectory-HMM: 95% confidence interval

Effect of MGE [Wu;'06]
Preference Score (%)
MGE

85.3

14.3

Conventional

Acoustic modeling research· Improve accuracy of p(O | W, λ)- Trended HMM [Dines;'01]- Polynomial segment model [Sun;'09]- Buried Markov model (BMM) [Bulyko;'02]- AR-HMM [Shannon;'09]- HSMM [Zen;'04]- Trajectory HMMs [Zen;'04]- Minimum generation error (MGE) training [Wu;'06]· Relax approximations- Bayesian approach [Hashimoto;'09]- MGE log-spectral distortion (MGE-LSD) [Wu;'08]- Statistical vocoder (STAVOCO) [Toda;'08]- Joint front-end/back-end training [Oura;'08]

Bayesian speech synthesis [Hashimoto;'09]λ̂ = arg max p(Ô | W, λ)λô = arg max p(o | w, λ̂)oô = arg maxoZ⇓p(o | w, λ)p(O | W, λ)p(λ)dλˆ)· ML estimation (point estimate, λ→ Bayesian estimation (distribution, p(λ | O ,W ))· Exact Bayesian is difficult due to integral over λ→ Variational Bayes (VB) [Beal;'03]

STAVOCO & MGE-LSD
λ̂ = arg max p(X, O | W, λ)λ= arg max p(X | O)p(O | W, λ)λ· STAVOCO [Toda;'08]- Developed for spectral analysis, but applicable to synthesis- O: (Mel) Cepstrum, X: harmonic components in log spectrum- O→X: stochastic linear transform· MGE-LSD [Wu;'08]- O: (Mel) LSP, X: harmonic components in log spectrum- O→X: deterministic non-linear mappingJoint front-end / back-end model training (1)Introduce context-dependent labels l & L
λ̂ = arg max
λô = arg max
o

X
L

X
l

p(O, L | W, λ)p(o, l | w, λ)Approximate summation by max
L̂ = arg max p(O, L | W) → λ̂ = arg max p(O | L̂, λ)Ll̂ = arg max p(l | w)lfront-end (word→label)λp(o | l̂, λ)→ ô = arg max
oback-end (training/synthesis)

Joint front-end / back-end model training (2)Introduce front-end model Λ to predict label from textL̂ = arg max p(O, L | W)

manual/automatic
annotationΛ̂ = arg max p(L̂ | W, Λ)

front-end modeltrainingl̂ = arg max p(l̂ | w, Λ)label predictionLΛl

1. Manually annotate prosodic labels of training data
2. Train prosodic label prediction models (e.g., decision trees)
3. Predict prosodic labels of input text to be synthesizedJoint front-end / back-end model training (3){λ̂, Λ̂} = arg max
λ,Λ

X
L

p(O | L, λ)p(L | W, Λ)

Joint front-end / back-end model training [Oura;08]- Label sequence: regarded as latent variable & marginalized→ Robust against labelling errors- Front-end & back-end models are simulteneously estimated→ Combine front-end/back-end models as a unified modelTime-line
16:15 ∼ 17:45: Second half
- Related topics* Unit selection & hybrid* Flexibility to control voice characteristics· Adaptation, interpolation, eigenvoice, multiple regress.

- Recent advances* Vocoding* Acoustic modeling* Over-smoothing compensation- Applications- Q&A (10min)

Over-smoothing problem· Speech parameter generation algorithm [Tokuda;'00]

Natural048Frequency (kHz)048Frequency (kHz)· Why?

Generated- Dynamic feature constraints make generated params. smooth- Sometimes too smooth ⇒ Sounds muffled- Details of spectral (formant) structure are removed- Use of advanced AM relaxes this problem, but not enoughCompensate over smoothing (1)· Emphasize spectral structure by post-filtering- Developed for speech coding- Reduce buzziness & muffling· Using training data explicitly in synthesis process- Conditional parameter generation algorithm [Masuko;'03]- Discrete HMM-based speech synthesis [Yu;'07]· Combining multiple-level statistics- Boosting-style additive F0 trees [Yao;'08]- DCT-based F0 models [Latorre;'08]- Intra-phoneme dynamics model [Tiomkin;'08]- Param. gen. algorithm considering global variance [Toda;'07]

Compensate over smoothing (2)Speech param. gen. algorithm considering GV [Toda;'07]
Generated

1

0
v(m)

2nd mel-cepstral coefficient

Natural

-1
0

1

2

3

Time [sec]Generated trajectory usually has smaller dynamic range

Compensate over smoothing (3)Speech param. gen. algorithm considering GV [Toda;'07]· Objective function of standard param. gen. algorithm
FML (c ; λ) = log N (W c ; µq , Σq )· Objective function of param. gen. algorithm considering GV
FGV (c ; λ, λGV ) = ω log N (W c ; µq , Σq ) + log N (v(c) ; µGV , ΣGV )- 1st term is the same as the above objective function- 2nd term works to keep dynamic range close to training data

⇒ Penalty for over-smoothingNatural048Frequency (kHz)Generated(standard)

048Frequency (kHz)Generated(w/ GV)

Compensate over smoothing (4)Effect of GV

048Frequency (kHz)Effect of GV [Zen;'07]

Average MOS

5

4.7

4.7

G

H

4
3.1

3
2
1

2.0

A

3.2

3.2

3.1

D

E

F

2.0

B

C

Spectra 24 mcep. 24 ST. 24 mcep. 24 ST. 39 ST. 24 mcep. 24 ST.
Model
HMM
HMM
HSMM HSMM HSMM HSMM HSMM
Par Gen non-GV non-GV non-GV non-GV non-GV
GV
GV

39 ST.
HSMM
GV

Compensate over smoothing (5)Incorporate GV into training· MGE + GV [Wu;07]ˆ = arg min {E (c; c̄q ) + ωE (v(c); v(c̄q ))}λλ· Trajectory HMM + GV [Toda;09]ˆ = arg max {log N (c; c̄q , Pq ) + ω log N (v(c); v(c̄q ), Σv )}λλTime-line
16:15 ∼ 17:45: Second half
- Related topics* Unit selection & hybrid* Flexibility to control voice characteristics· Adaptation, interpolation, eigenvoice, multiple regress.

- Recent advances* Vocoding* Acoustic modeling* Over-smoothing compensation- Applications- Q&A (10min)

Singing voice synthesis [Saino;'06]
Singing voice signal
Singing voice
database

Spectral parameter
extraction

Excitation parameter
extraction

Mel-cepstrum

F0
Training of HMMs

Training part
HMMs
(spectrum & F0)

Duration models

Musical score...

Conversion...
A song HMM

LabelParameter generationfrom the song HMM

F0Synthesis part

Excitation
generation

Mel-cepstrum
MLSA filter

Synthesized
singing voiceEmotional speech synthesis [Tsuzuki;04]"Finally, I present a conclusion. We constructed
emotional speech synthesizer based on
HMM-based speech synthesis system.
Furthermore, a modeling technique based on
subjective evaluation was proposed.
Listening test results show that the proposed
technique improves the expressiveness ofemotions for synthesized speech, particularly,
neutral and happiness.
Future work includes constructing more effective
contexts for emotional speech synthesis.
In addition, improvements in the quality ofsynthesized speech is also future work. That's all."neutralangry

neutralangry

Audio-visual synthesis (pixel-based) [Sako;'00]
Pixel image: high dimensionality⇒ Dimensionality reduction by PCA
PCA coefficients

Eigenlips

Real image
Reconstructedimage

Audio-visual synthesis (model-based) [Tamura;'98]yu0 u1 u2l0 l1 l2w

u3
l3

Human motion synthesis· Record human movements by a motion capture system· Train HMMs

· Synthesize motions
Finger [Haoka;'02]

Walking [Niwase;05]

Head [Hofer;'07]

Online handwriting recognition [Ma;'07]
VoicedUnvoicedΩ1 = R 1
N1 (x)Ω2 = RΩ1 = R 2
N1 (x)

Imaginary stroke(pen up)Ω2 = R 0

Log Frequency

0Real stroke(pen down)Time· Observation consists of continuous value or discrete symbol⇒ Can be modeled by MSD-HMMs· Properly handle imaginary strokes⇒ Better discrimination between similar characters

Very small footprint synthesizer [Morioka;'04]· Acoustic model size < 100KBytes- Tree size

- Order of mel-cepstral analysis- Variance flooring scale· 0.1 x Real Time on a old desktop PC
Samples

File size *Before shrinking

950 KB

After shrinking

100 KB* In single precision floating point

Summary (second half)
Recent advances in HMM-based speech synthesis· Related topics- Unit selection synthesis & hybrid- Flexibility to control voice characteristics· Recent advances- Vocoding- Acoustic modeling- Over-smoothing compensation· Applications

Further information
HTS website: http://hts.sp.nitech.ac.jp/· hts-users mailing list

- Over 500 posts per year
- All posts are archived & searchable- Bug reports, Q&A, announce· Download page

- Patches/fixes after official releases· Publication page

- List of important papers- Tutorial slides· Demo page

- List of demonstrations

Acknowledgments / Special thanks!!
K. Tokuda; Overall concept & design
H. Zen; Throwing away all old codes of HTS & rewriting them as HTS ver. 1.0--2.1. Former maintainer of HTS
K. Oura; Implementation of hts_engine API. Current maintainer of HTS
J. Yamagishi; Testing out various adaptation algorithms. Implementation of CSMAPLR adaptation
T. Yoshimura; Development of the first version of HTS
M. Tamura; Implementation of various speaker adaptation algorithms for HMM-based speech synthesis
S. Sako; Applying HMM-based speech synthesis to audio-visual synthesis
N. Miyazaki; Implementation of the first version of MSD-HMM trainer
T. Yamada; Testing out various HMM training algorithms for HMM-based speech synthesis
T. Masuko; Discussions in detail & implementing speech parameter generation algorithm
K. Koishida; Discussions in detail & implementing speech coder based on mel-generalized cepstral analysis
Y. Nankaku; Discussions & supervising students working on HMM-based speech synthesis
T. Kobayashi; Discussions & supervising students working on HMM-based speech synthesis
T. Kitamura; Discussions & supervising students on HMM-based speech synthesis
S. Imai; Supervising students working on HMM-based speech synthesis
A. W. Black; Discussions about all aspects of speech synthesis, helping to use Festival
T. Toda; Discussions & implementation of speech parameter generation algorithm considering GV
Y.-J. Wu; Discussions & implementation of minimum generation error training
T. Nose; Implementing multiple-regression based voice characteristics control techniques
M. Tachibana; Testing out various speaking style modification techniques
S. J. Young, P. C. Woodland, M. J. F. Gales, G. Evermann, & other HTK developers; Development of HTK& others,...

References (1)Sagisaka;'92 - "ATR nu-TALK speech synthesis system," ICSLP, '92.
Black;'96 - "Automatically clustering similar units...," Eurospeech, '97.
Beutnagel;'99 - "The AT&T Next-Gen TTS system," Joint ASA, EAA, & DAEA meeting, '99.
Yoshimura;'99 - "Simultaneous modeling of spectrum ...," Eurospeech, '99.
Itakura;'70 - "A statistical method for estimation of speech spectral density...," Trans. IEICE, J53-A, '70.
Imai;'88 - "Unbiased estimator of log spectrum and its application to speech signal...," EURASIP, '88.
Kobayashi;'84 - "Spectral analysis using generalized cepstrum," IEEE Trans. ASSP, 32, '84.
Tokuda;'94 - "Mel-generalized cepstral analysis -- A unified approach to speech spectral...," ICSLP, '94.
Imai;'83 - "Cepstral analysis synthesis on the mel frequency scale," ICASSP, '83.
Fukada;'92 - "An adaptive algorithm for mel-cepstral analysis of speech," ICASSP, '92.
Itakura;'75 - "Line spectrum representation of linear predictive coefficients of speech...," J. ASA (57), '75.
Tokuda;'02 - "Multi-space probability distribution HMM," IEICE Trans. E85-D(3), '02.
Odell;'95 - "The use of context in large vocaburary...," PhD thesis, University of Cambridge, '95.
Shinoda;'00 - "MDL-based context-dependent subword modeling...," Journal of ASJ(E) 21(2), '00.
Yoshimura;'98 - "Duration modeling for HMM-based speech synthesis," ICSLP, '98.
Tokuda;'00 - "Speech parameter generation algorithms for HMM-based speech synthesis," ICASSP, '00.
Kobayashi;'85 - "Mel generalized-log spectrum approximation...," IEICE Trans. J68-A (6), '85.
Hunt;'96 - "Unit selection in a concatenative speech synthesis system using...," ICASSP, '96.
Donovan;'95 - "Improvements in an HMM-based speech synthesiser," Eurospeech, '95.
Kawai;'04 - "XIMERA: A new TTS from ATR based on corpus-based technologies," ISCA SSW5, '04.
Hirai;'04 - "Using 5 ms segments in concatenative speech synthesis," Proc. ISCA SSW5, '04.

References (2)Rouibia;'05 - "Unit selection for speech synthesis based on a new acoustic target cost," Interspeech, '05.
Huang;'96 - "Whistler: A trainable text-to-speech system," ICSLP, '96.
Mizutani;'02 - "Concatenative speech synthesis based on HMM," ASJ autumn meeting, '02.
Ling;'07 - "The USTC and iFlytek speech synthesis systems...," Blizzard Challenge workshop, 07.
Ling;'08 - "Minimum unit selection error training for HMM-based unit selection...," ICASSP, 08.
Plumpe;'98 - "HMM-based smoothing for concatenative speech synthesis," ICSLP, '98.
Wouters;'00 - "Unit fusion for concatenative speech synthesis," ICSLP, '00.
Okubo;'06 - "Hybrid voice conversion of unit selection and generation...," IEICE Trans. E89-D(11), '06.
Aylett;'08 - "Combining statistical parametric speech synthesis and unit selection..." LangTech, '08.
Pollet;'08 - "Synthesis by generation and concatenation of multiform segments," Interspeech, '08.
Yamagishi;'06 - "Average-voice-based speech synthesis," PhD thesis, Tokyo Inst. of Tech., '06.
Yoshimura;'97 - "Speaker interpolation in HMM-based speech synthesis system," Eurospeech, '97.
Tachibana;'05 - "Speech synthesis with various emotional expressions...," IEICE Trans. E88-D(11), '05.
Kuhn;'00 - "Rapid speaker adaptation in eigenvoice space," IEEE Trans. SAP 8(6), '00.
Shichiri;'02 - "Eigenvoices for HMM-based speech synthesis," ICSLP, '02.
Fujinaga;'01 - "Multiple-regression hidden Markov model," ICASSP, '01.
Nose;'07 - "A style control technique for HMM-based expressive speech...," IEICE Trans. E90-D(9), '07.
Yoshimura;'01 - "Mixed excitation for HMM-based speech synthesis," Eurospeech, '01.
Kawahara;'97 - "Restructuring speech representations using a ...", Speech Communication, 27(3), '97.
Zen;'07 - "Details of the Nitech HMM-based speech synthesis system...", IEICE Trans. E90-D(1), '07.
Abdl-Hamid;'06 - "Improving Arabic HMM-based speech synthesis quality," Interspeech, '06.

References (3)Hemptinne;'06 - "Integration of the harmonic plus noise model into the...," Master thesis, IDIAP, '06.
Banos;'08 - "Flexible harmonic/stochastic modeling...," V. Jornadas en Tecnologias del Habla, '08.
Cabral;'07 - "Towards an improved modeling of the glottal source in...," ISCA SSW6, '07.
Maia;'07 - "An excitation model for HMM-based speech synthesis based on ...," ISCA SSW6, '07.
Ratio;'08 - "HMM-based Finnish text-to-speech system utilizing glottal inverse filtering," Interspeech, '08.
Drugman;'09 - "Using a pitch-synchronous residual codebook for hybrid HMM/frame...", ICASSP, '09.
Dines;'01 - "Trainable speech synthesis with trendedn hidden Markov models," ICASSP, '01.
Sun;'09 - "Polynomial segment model based statistical parametric speech synthesis...," ICASSP, '09.
Bulyko;'02 - "Robust splicing costs and efficient search with BMM models for...," ICASSP, '02.
Shannon;'09 - "Autoregressive HMMs for speech synthesis," Interspeech, '09.
Zen;'06 - "Reformulating the HMM as a trajectory model...", Computer Speech & Language, 21(1), '06.
Wu;'06 - "Minimum generation error training for HMM-based speech synthesis," ICASSP, '06.
Hashimoto;'09 - "A Bayesian approach to HMM-based speech synthesis," ICASSP, '09.
Wu;'08 - "Minimum generation error training with log spectral distortion for...," Interspeech, '08.
Toda;'08 - "Statistical approach to vocal tract transfer function estimation based on...," ICASSP, '08.
Oura;'08 - "Simultaneous acoustic, prosodic, and phrasing model training for TTS...," ISCSLP, '08.
Ferguson;'80 - "Variable duration models...," Symposium on the application of HMM to text speech, '80.
Levinson;'86 - "Continuously variable duration hidden...," Computer Speech & Language, 1(1), '86.
Beal;'03 - "Variational algorithms for approximate Bayesian inference," PhD thesis, Univ. of London, '03.
Masuko;'03 - "A study on conditional parameter generation from HMM...," Autumn meeging of ASJ, '03.
Yu;'07 - "A novel HMM-based TTS system using both continuous HMMs and discrete...," ICASSP, '07.

References (4)Qian;'08 - "Generating natural F0 trajectory with additive trees," Interspeech, '08.
Latorre;'08 - "Multilevel parametric-base F0 model for speech synthesis," Interspeech, '08.
Tiomkin;'08 - "Statistical text-to-speech synthesis with improved dynamics," Interspeech, '08.
Toda;'07 - "A speech parameter generation algorithm considering global...," IEICE Trans. E90-D(5), '07.
Wu;'08 - "Minimum generation error criterion considering global/local variance...," ICASSP, '08.
Toda;'09 - "Trajectory training considering global variance for HMM-based speech...," ICASSP, '09.
Saino;'06 - "An HMM-based singing voice synthesis system," Interspeech, '06.
Tsuzuki;'04 - "Constructing emotional speech synthesizers with limited speech...," Interspeech, '04.
Sako;'00 - "HMM-based text-to-audio-visual speech synthesis," ICSLP, '00.
Tamura;'98 - "Text-to-audio-visual speech synthesis based on parameter generation...," ICASSP, '98.
Haoka;'02 - "HMM-based synthesis of hand-gesture animation," IEICE Commun. syst.,102(517), '02.
Niwase;'05 - "Human walking motion synthesis with desired pace and...," IEICE Trans. E88-D(11), '05.
Hofer;'07 - "Speech driven head motion synthesis based on a trajectory model," SIGGRAPH, '07.
Ma;'07 - "A MSD-HMM approach to pen trajectory modeling for online handwriting...," ICDAR, '07.
Morioka;'04 - "Miniaturization of HMM-based speech synthesis," Autumn meeting of ASJ, '04.
Kim;'06 - "HMM-Based Korean speech synthesis system for...," IEEE Trans. Comsum. Elec., 52(4), '06.
Klatt;'82 - "The Klatt-Talk text-to-speech system," ICASSP, '82.