SPEECH PARAMETER GENERATION ALGORITHMS FOR
HMM-BASED SPEECH SYNTHESIS
Keiichi Tokuda 1 , Takayoshi Yoshimura 1 , Takashi Masuko 2 , Takao Kobayashi 2 , Tadashi Kitamura 1 ,1

2

Department of Computer Science, Nagoya Institute of Technology, Nagoya, 466-8555 Japan
Interdisciplinary Graduate School of Science and Engineering, Tokyo Institute of Technology, Yokohama, 226-8502 Japan
Email: {tokuda,yossie,kitamura}@ics.nitech.ac.jp, {masuko,tkobayas}@ip.titech.ac.jp

ABSTRACTThis paper derives a speech parameter generation algorithm for
HMM-based speech synthesis, in which speech parameter sequenceis generated from HMMs whose observation vector consists ofspectral parameter vector and its dynamic feature vectors. In thealgorithm, we assume that the state sequence (state and mixture sequence for the multi-mixture case) or a part of the state sequence
is unobservable (i.e., hidden or latent). As a result, the algorithm
iterates the forward-backward algorithm and the parameter generation algorithm for the case where state sequence is given. Experimental results show that by using the algorithm, we can reproduce
clear formant structure from multi-mixture HMMs as comparedwith that produced from single-mixture HMMs.
1. INTRODUCTION
The increasing availability of large speech databases makes it possible to construct speech synthesis systems, which are referred toas data-driven or corpus-based approach, by applying statisticallearning algorithms. These systems, which can be automaticallytrained, not only generate natural and high quality synthetic speech
but also can reproduce voice characteristics of the original speaker.
For constructing such a system, the use of hidden Markovmodels (HMMs) has arisen largely. HMMs have successfully been
applied to modeling the sequence of speech spectra in speech recognition systems, and the performance of HMM-based speech recognition systems have been improved by techniques which utilize theflexibility of HMMs: context-dependent modeling, dynamic feature parameters, mixtures of Gaussian densities, tying mechanism,speaker and environment adaptation techniques. HMM-based approaches to speech synthesis can be categorized as follows:
1. Transcription and segmentation of speech database [1].
2. Construction of inventory of speech segments [2], [3].
3. Run-time selection of multiple instances of speech segments[4], [5].
4. Speech synthesis from HMMs themselves [6]–[9].

voice characteristics of synthetic speech can be changed by transforming HMM parameters appropriately. From this point of view,
we have proposed parameter generation algorithms [10], [11] for
HMM-based speech synthesis, and constructed a speech synthesis
system [8], [9]. Actually, we have shown that we can change voice
characteristics of synthetic speech by applying a speaker adaptation technique [12], [13] or a speaker interpolation technique [14].
The main feature of the system is the use of dynamic feature: byinclusion of dynamic coefficients in the feature vector, the dynamic
coefficients of the speech parameter sequence generated in synthesis are constrained to be realistic, as defined by the parameters ofthe HMMs.
This paper describes algorithms for speech parameter generation from HMMs. In the previously proposed algorithms [10], [11]for speech parameter generation, state sequence is assumed to begiven, or determined based on a maximum likelihood criterion. Inthis paper, we derive a new algorithm in which we assume that thestate sequence (state and mixture sequence for the multi-mixturecase) or a part of the state sequence is unobservable (i.e., hidden
or latent), and show examples of speech spectra generated by theproposed algorithm.
The rest of this paper is organized as follows. Section 2 summarizes the previously proposed algorithms for speech parameter
generation, and derives a new algorithm. Experimental results areshown in Section 3. Concluding remarks are presented in the finalsection.
2. SPEECH PARAMETER GENERATION BASED ON
MAXIMUM LIKELIHOOD CRITERION
For a given continuous mixture HMM λ, we derive an algorithm
for determining speech parameter vector sequenceO = o
1 , o2 , . . . , oT



(1)

in such a way that
P (O|λ) =

P (O, Q|λ)

(2)

all Q

In approaches 1–3, by using a waveform concatenation algorithm,e.g., PSOLA algorithm, a high quality synthetic speech could beproduced. However, to obtain various voice characteristics, large
amounts of speech data are necessary, and it is difficult to collect,segment, and store these data. On the other hand, in approach 4,

is maximized with respect to O, whereThis work was partially supported by the Ministry of Education, Science, Sports and Culture Japan, Grant-in-Aid for Scientific Research (B) 2,1055125, 1998, Encouragement of Young Scientists (A), 10780226, 1998.

is the state and mixture sequence, i.e., (q, i) indicates the i-th mixture of state q. We assume that the speech parameter vector ot consists of the static feature vector ct = [ct (1), ct (2), . . . , ct (M )]

Q = {(q1 , i1 ), (q2 , i2 ), . . . , (qT , iT )}(3)

(e.g., cepstral coefficients) and dynamic feature vectors ∆ct , ∆2 ct(e.g., delta and delta-delta cepstral coefficients, respectively), that2  is, ot = [ct , ∆ct , ∆ ct ] , where the dynamic feature vectors
are calculated by(0)(0)

L− = L+ = 0, and w(0) (0) = 1. Under the condition (9),maximizing P (O|Q, λ) with respect to O is equivalent to thatwith respect to C. By setting∂ log P (W C|Q, λ)= 0,∂C

(1)

L+



=

∆ctw

(1)

(τ )ct+τ(4)(1)
τ =−L−

we obtain a set of equations
W  U −1 W C = W  U −1 M  .

(2)

L+

∆2 c t=w(2) (τ )ct+τ .

(5)

(2)
τ =−L−We have derived algorithms [10], [11] for solving the following problems:Case 1. For given λ and Q, maximize P (O|Q, λ) with respect
to O under the conditions (4), (5).
Case 2. For a given λ, maximize P (O, Q|λ) with respect to Q
and O under the conditions (4), (5).
In this section, we will review the above algorithms and derive analgorithm for the problem:
Case 3. For a given λ, maximize P (O|λ) with respect to O under the conditions (4), (5).
2.1. Case 1: Maximizing P (O|Q, λ) with respect to O
First, consider maximizing P (O|Q, λ) with respect to O for a
fixed state and mixture sequence Q. The logarithm of P (O|Q, λ)can be written as
1log P (O|Q, λ) = − O  U −1 O + O  U −1 M + K
2
whereU −1=M=−1−1diag U −1
q1 ,i1 , U q2 ,i2 , ..., U qT ,iT µq1 ,i1 , µ
q2 ,i2 , ..., µqT ,iT(6)2.2. Case 2: Maximizing P (O, Q|λ) with respect to O and Q
This problem can be solved by evaluating maxC P (O, Q|λ) =maxC P (O|Q, λ)P (Q|λ) for all Q. However, it is impracticalbecause there are too many combinations of Q. We have developed a fast algorithm for searching for the optimal or sub-optimal
state sequence keeping C optimal in the sense that P (O|Q, λ) is
maximized with respect to C [10], [11].
To control temporal structure of speech parameter sequence
appropriately, HMMs should incorporate state duration densities.
The probability P (O, Q|λ) can be written as P (O, Q|λ) =P (O, i|q, λ)P (q|λ), where q = {q1 , q2 , . . . , qT }, i = {i1 , i2 ,. . . , iT }, and the state duration probability P (q|λ) is given byN

log pqn (dqn )

(16)

(7)

n=1

(8)

where the total number of states which have been visited duringT frames is N , and pqn (dqn ) is the probability of dqn consecutive observations in state qn . If we determine the state sequence q
only by P (q|λ) independently of O, maximizing P (O, Q|λ) =P (O, i|q, λ)P (q|λ) with respect to O and Q is equivalent tomaximizing P (O, i|q, λ) with respect to O and i. Furthermore, if
we assume that state output probabilities are single-Gaussian, i isunique. Therefore, the solution is obtained by solving (15) in thesame way as the Case 1.

µqt , it and U qt , it are the 3M × 1 mean vector and the 3M × 3M
covariance matrix, respectively, associated with it -th mixture ofstate qt , and the constant K is independent of O.
It is obvious that P (O|Q, λ) is maximized when O = M
without the conditions (4), (5), that is, the speech parameter vector
sequence becomes a sequence of the mean vectors. Conditions (4),(5) can be arranged in a matrix form:
O = WC(15)

For direct solution of (15), we need O(T 3 M 3 ) operations1 because
W  U −1 W is a T M ×T M matrix. By utilizing the special structure of W  U −1 W , (15) can be solved by the Cholesky decomposition or the QR decomposition with O(T M 3 L2 ) operations2 ,(n)where L = maxn∈{1,2},s∈{−,+} Ls . Equation (15) can also besolved by an algorithm derived in [10], [11], which can operate ina time-recursive manner [16].

log P (q|λ) =(14)(9)
2.3. Case 3: Maximizing P (O|λ) with respect to O

whereC=[c1 , c2 , . . . , cT ]

W=[w 1 , w 2 , . . . , w T ]

=(0)
(1)
(2)
wt , wt , wt

wt
(n)
wt

=



(10)
(11)

(12)

0M ×M , . . . , 0M ×M , w(n)

1stQ(O, O  ) =(n)(−L− )I M ×M ,(n). . . , w(n) (0)I M ×M , . . . , w(n) (L+ )I M ×M ,(n)0M ×M , . . . , 0M ×M
T -th



(t+L+ )-th,n = 0, 1, 2



P (O, Q|λ) log P (O  , Q|λ).

(17)

all Q(n)(t−L− )-tht-th

We derive an algorithm based on an EM algorithm, which finda critical point of the likelihood function P (O|λ). An auxiliary
function of current parameter vector sequence O and new parameter vector sequence O  is defined by

1 When U
3q,i is diagonal, it is reduced to O(T M ) since each of theM -dimensions can be calculated independently.
2 When U
2q,i is diagonal, it is reduced to O(T M L ). Furthermore,
(1)

(13)

(1)

when L− = −1, L+ = 0, and w(2) (i) ≡ 0, it is reduced to O(T M )as described in [15].

(kHz)

02468(kHz)02468

1-mixture(kHz)

2-mixture02468(kHz)

4-mixture024688-mixture
Figure 1: Generated spectra for a sentence fragment “kiNzokuhiroo.”

It can be shown that by substituting O  which maximizes Q(O, O  )for O, the likelihood increases unless O is a critical point of thelikelihood. Equation (17) can be written as

1Q(O, O ) = P (O|λ) − O  U −1 O  + O  U −1 M + K
2(18)whereU −1=U −1t

=



−1−1
diag U −1
1 ,U2 ,..., UTγt (q, i)U −1q,i(20)q,iU −1 MU −1t µt==





−1−1
U −1
1 µ1 , U 2 µ2 , . . . , U T µT



(19)

γt (q, i) U −1q,i µq,i (21)(22)q,iand the constant K is independent of O  . The occupancy probability γt (q, i) defined byγt (q, i) = P (qt = (q, i)|O, λ)(23)

can be calculated with the forward-backward inductive procedure.
Under the condition O  = W C  , C  which maximizes Q(O, O  )is given by the following set of equations:W  U −1 W C  = W  U −1 M .

(24)

The above set of equations has the same form as (15). Accordingly,
it can be solved by the algorithm for solving (15).
The whole procedure is summarized as follows:

Step 0. Choose an initial parameter vector sequence C.
Step 1. Calculate γt (q, i) with the forward-backward algorithm.
Step 2. Calculate U −1 and U −1 M by (19)–(22), and solve (24).
Step 3. Set C = C  . If a certain convergence condition is satisfied, stop; otherwise, goto Step 1.
From the same reason as Case 2, HMMs should incorporate
state duration densities. If we determine the state sequence q onlyby P (q|λ) independently of O in a manner similar to the previoussection, only the mixture sequence i is assumed to be unobservable3 . Further, we can also assume that Q is unobservable but
phoneme or syllable durations are given.
3. EXAMPLE
We used phonetically balanced 450 sentences from ATR Japanese
speech database for training. Speech signal were sampled at 16
kHz and windowed by a 25.6-ms Blackman window with a 5ms shift, and then mel-cepstral coefficients were obtained by amel-cepstral analysis technique [18]. Feature vector consists of
25 mel-cepstral coefficients including the zeroth coefficient, theirdelta and delta-delta coefficients. We used 5-state left-to-right
HMMs. Decision-tree based state clustering [19] was applied tothe context-dependent phoneme model set, and the resultant HMM
set has approximately 900 states. We assume that the state andmixture sequence is unobservable but phoneme durations are given
from phoneme duration densities in a manner similar to [20]. It
has found that a few iterations are sufficient for convergence of the3 For this problem, an algorithm based on a direct search has also been
proposed in [17].

Log magnitude (dB)

100
mix1
mix8

80
60
40
20

0246Frequency (kHz)

8Figure 2: Spectra obtained from 1-mixture HMMs and 8-mixture
HMMs.
proposed algorithm. Fig. 1 shows generated spectra for a Japanese
sentence fragment “kiNzokuhiroo” taken from a sentence which is
not included in the training data. Fig. 2 compares two spectra obtained from single-mixure HMMs and 8-mixture HMMs, respectively, for the same temporal position of the sentence fragment.
It is seen from Fig. 1 and Fig. 2 that with increasing mixtures,the formant structure of the generated spectra get clearer. From
informal listening of the synthetic speech, it has been observedthat the quality of the synthetic speech is considerably improvedby increasing mixtures.
When we use single-mixture HMMs, the formant structureof spectrum corresponding to each mean vector µq,i might bevague since µq,i is the average of different speech spectra. One
can increase the number of decision tree leaf clusters. However,it might result in perceivable discontinuities in synthetic speechsince overly large tree will be overspecialized to training data andgeneralized poorly. We expect that the proposed algorithm can
avoid this situation in a simple manner.
4. CONCLUSION
This paper has derived an algorithm for speech parameter generation from HMM whose observation vector consists of spectral parameter vector and its dynamic feature vectors. In the algorithm,
we assume that the state sequence (state and mixture sequence
for the multi-mixture case) or a part of the state sequence is unobservable (i.e., hidden or latent). The algorithm is based on anEM algorithm. As a result, it iterates the forward-backward algorithm and the parameter generation algorithm for the case wherestate sequence is given. Experimental results have shown thatby using the algorithm, we can reproduce clear formant structurefrom multi-mixture HMMs as compared with that produced fromsingle-mixture HMMs. Results of subjective evaluation tests ofthe synthetic speech will be presented in the near future.
5. REFERENCES[1] A. Ljolje, J. Hirschberg and J. P. H. van Santen, “Automaticspeech segmentation for concatenative inventory selection,”in Progress in Speech Synthesis, ed. J. P. H. van Santen, R.
W. Sproat, J. P. Olive and J. Hirschberg, Springer-Verlag,
New York, 1997.
[2] R. E. Donovan and P. C. Woodland, “Automatic speechsynthesiser parameter estimation using HMMs,” in Proc.
ICASSP, 1995, pp.640–643.

[3] X. Huang, A. Acero, H. Hon, Y. Ju, J. Liu, S. Meredith andM. Plumpe, “Recent improvements on Microsoft’s trainabletext-to-speech system -Whistler,” in Proc. ICASSP, 1997,
pp.959–962.
[4] H. Hon, A. Acero, X. Huang, J. Liu and M. Plumpe, “Automatic generation of synthesis units for trainable text-tospeech synthesis,” in Proc. ICASSP, 1998, pp.293–306.
[5] R. E. Donovan and E. M. Eide, “The IBM Trainable Speech
Synthesis System,” in Proc. ICSLP, 1998, pp.1703–1706.
[6] A. Falaschi, M. Giustiniani and M. Verola, “A hidden
Markov model approach to speech synthesis,” in Proc. EUROSPEECH, 1989, pp.187–190.
[7] M. Giustiniani and P. Pierucci, “Phonetic ergodic HMM forspeech synthesis,” in Proc. EUROSPEECH, 1991, pp.349–352.
[8] T. Masuko, K. Tokuda, T. Kobayashi and S. Imai, “Speechsynthesis from HMMs using dynamic features,” in Proc.
ICASSP, 1996, pp.389–392.
[9] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi and T.
Kitamura, “Simultaneous modeling of spectrum, pitch andduration in HMM-based speech synthesis,” in Proc. EUROSPEECH, 1999, pp.2347–2350.
[10] K. Tokuda, T. Kobayashi and S. Imai, “Speech parameter
generation from HMM using dynamic features,” in Proc.
ICASSP, 1995, pp.660–663.
[11] K. Tokuda, T. Masuko, T. Yamada, T. Kobayashi and S. Imai,“An algorithm for speech parameter generation from continuous mixture HMMs with dynamic features,” in Proc. EUROSPEECH, 1995, pp.757–760.
[12] T. Masuko, K. Tokuda, T. Kobayashi and S. Imai, “Voicecharacteristics conversion for HMM-based speech synthesis
system,” in Proc. ICASSP, 1997, pp.1611–1614.
[13] M. Tamura, T. Masuko, K. Tokuda and T. Kobayashi,”Speaker adaptation for HMM-based speech synthesis system using MLLR,” in Proc. ESCA/COCOSDA Third International Workshop on Speech Synthesis, 1998, pp.273–276.
[14] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi and T.
Kitamura, “Speaker Interpolation in HMM-Based Speech
Synthesis System,” in Proc. EUROSPEECH, 1997, pp.2523–2526.
[15] A. Acero, “Formant analysis and synthesis using hidden
Markov models,” in Proc. EUROSPEECH, 1999, pp.1047–1050.
[16] K. Koishida, K. Tokuda, T. Masuko and T. Kobayashi, “Vector quantization of speech spectral parameters using statisticsof dynamic features,” in Proc. ICSP, 1997, pp.247–252.
[17] W. Tachiwa and S. Furui, “A study of speech synthesis usingHMMs,” in Proc. Spring Meeting of Acoustical Society ofJapan, Mar. 1999, pp.239–240 (in Japanese).
[18] T. Fukada, K. Tokuda, T. Kobayashi and S. Imai, “An adaptive algorithm for mel-cepstral analysis of speech,” in Proc.
ICASSP, 1992, pp.137–140.
[19] J. J. Odell, “The use of context in large vocabulary speech
recognition,” PhD thesis, Cambridge University, 1995.
[20] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi and T.
Kitamura, ”Duration modeling for HMM-based speech synthesis,” in Proc. ICSLP, 1998, pp.29–32.