SIMULTANEOUS MODELING OF SPECTRUM, PITCH AND DURATIONIN HMM-BASED SPEECH SYNTHESISTakayoshi Yoshimuray, Keiichi Tokuday, Takashi Masukoyy, Takao Kobayashiyy and Tadashi KitamurayyNagoya Institute of Technology, Gokiso, Shouwa-ku, Nagoya, 466-8555 Japan
yyTokyo Institute of Technology, Nagatsuta, Midori-ku, Yokohama, 226-8502 Japan
E-mail: fyossie, tokuda, kitamurag@ics.nitech.ac.jp, fmasuko, tkobayasg@ip.titech.ac.jp

ABSTRACTIn this paper, we describe an HMM-based speech synthesis system in which spectrum, pitch and state duration
are modeled simultaneously in a unified framework of HMM. In the system, pitch and state duration are modeledby multi-space probability distribution HMMs and multi-dimensional Gaussian distributions, respectively. The
distributions for spectral parameter, pitch parameter andthe state duration are clustered independently by using adecision-tree based context clustering technique. Synthetic
speech is generated by using an speech parameter generation algorithm from HMM and a mel-cepstrum based vocoding technique. Through informal listening tests, we haveconfirmed that the proposed system successfully synthesizes natural-sounding speech which resembles the speakerin the training database.

1.

INTRODUCTION

Although most text-to-speech synthesis systems can synthesize speech with acceptable quality, they still cannot
synthesize speech with various voice characteristics suchas speaker individualities and emotions. To obtain various voice characteristics in text-to-speech synthesis systems
based on the selection and concatenation of acoustical units, a large amount of speech data is necessary. However,
it is difficult to collect, segment, and store it. From thesepoints of view, in order to construct speech synthesis system
which can generate various voice characteristics, we haveproposed an HMM-based speech synthesis system [1].
Several HMM-based concatenative speech synthesis systems have also been proposed (e.g., [2], [3]). Our system,
however, differs from them in that our system generates synthetic speech from HMMs themselves by using a speech parameter generation algorithm [4] and a mel-cepstrum based
vocoding technique [5], [6]. In the parameter generation
algorithm, by the inclusion of dynamic coefficients in thefeature vector, the dynamic coefficients of the speech parameter sequence generated in synthesis are constrained tobe realistic, as defined by the parameters of the HMMs. By transforming HMM parameters appropriately, voice
characteristics of synthetic speech can be changed since thesystem generates speech from the HMMs. In fact, we haveshown that we can change voice characteristics of synthetic
speech by applying a speaker adaptation technique [7] ora speaker interpolation technique [8]. However, to change
not only speaker individuality but also speaking style suchas emotion expression, pitch and duration parameters havec
Stream 1(Spectrum part)∆cContinuous probability distribution

2∆ cStream 2(Pitch part)

pδp2

δ p

Multi-space probability distribution
Multi-space probability distribution
Multi-space probability distribution

Figure 1. Feature vector.
to be appropriately transformed.
In this paper, in order to apply speaker adaptation/interpolation techniques to spectrum, pitch and state duration, simultaneously, and to synthesize speech with various
voice characteristics, we construct a speech synthesis system in which spectrum, pitch and state duration are modeled
simultaneously in a unified framework of HMM. In the system, pitch and state duration are modeled by multi-spaceprobability distribution HMMs [9] and multi-dimensional
Gaussian distributions [10], respectively. The feature vector of HMMs used in the system consists of two streams,i.e., the one for spectral parameter vector and the other for
pitch parameter vector, and each phoneme HMM has its
state duration densities. The distributions for spectral parameter, pitch parameter and state duration are clusteredindependently by using a decision-tree based context clustering technique [11]. In the context clustering, we takeaccount of contextual factors which affect spectrum, pitchand duration such as phone identity factors, stress-related
factors and locational factors.
The rest of this paper is structured as follows. Section
2 describes simultaneous modeling of spectrum, pitch andstate duration. Section 3 describes the proposed text-tospeech synthesis system. Experimental results are presented in Section 4, and concluding remarks and our plans for
future work are presented in the final section.
2. SIMULTANEOUS MODELING
2.1. Spectrum and Pitch ModelWe use mel-cepstral coefficients as spectral parameter. Sequences of mel-cepstral coefficient vector, which are obtained from speech database using a mel-cepstral analysis
technique [5], are modeled by continuous density HMMs.
The mel-cepstral analysis technique enables speech to be
re-synthesized from the mel-cepstral coefficients using theMLSA (Mel Log Spectrum Approximation) filter [6].

Pitch patterns are modeled by a hidden Markov model
based on multi-space probability distribution (MSD-HMM)[9]. We cannot apply the conventional discrete or continuous HMMs to pitch pattern modeling since the observation
sequence of pitch pattern is composed of one-dimensional
continuous values and a discrete symbol which represents“unvoiced”. The MSD-HMM includes discrete HMM andcontinuous mixture HMM as special cases, and further canmodel the sequence of observation vectors with variable dimension including zero-dimensional observations, i.e., discrete symbols. As a result, MSD-HMMs can model pitch
patterns without heuristic assumption.
We construct spectrum and pitch models by using embedded training because the embedded training does notneed label boundaries when appropriate initial models areavailable. However, if spectrum models and pitch models are embedded-trained separately, speech segmentations
may be discrepant between them.
To avoid this problem, context dependent HMMs aretrained with feature vector which consists of spectrum, pitch
and their dynamic features (Fig. 1).
2.2. State Duration Model
State duration densities are modeled by single Gaussian
distributions [10]. Dimension of state duration densities
is equal to the number of state of HMM, and the n-th dimension of state duration densities is corresponding to then-th state of HMMs1 . Since state durations are modeledby continuous distributions, our approach has the following
advantages:

 The speaking rate of synthetic speech can be varied
easily.
 There is no need for label boundaries when appropriate
initial models are available since the state duration
densities are estimated in the embedded training stageof phoneme HMMs.
There have been proposed techniques for training HMMs
and their state duration densities simultaneously (e.g., [12]).
However, these techniques require a large storage and computational load. In this paper, state duration densities areestimated by using state occupancy probabilities which are
obtained in the last iteration of embedded re-estimation
[10].
3.

CONTEXT DEPENDENT MODEL3.1. Contextual Factors
There are many contextual factors (e.g., phone identity factors, stress-related factors, locational factors) that affectspectrum, pitch and duration. In this paper, following contextual factors are taken into account: mora2 count of sentence position of breath group in sentence mora count of fpreceding, current, succeedingg breathgroup
 position of current accentual phrase in current breathgroup
 mora count and accent type of fpreceding, current,succeedingg accentual phrase
1 We assume the left-to-right model with no skip.
2 A mora is a syllable-sized unit in Japanese.

State Duration
Model
HMM
for Spectrum
and Pitch

S1

S2

S3

Decision Treefor
Spectrum

Decision Treefor
State Duration Model

Decision Treefor
Pitch

Figure 2. Decision trees.
Context Dependent
Duration Models
Context Dependent
HMMsTEXTSynthesis

Context-basedLabel Sequence

State Duration
Densities
Sentence
HMM
State Duration

d1d2Pitch

p1 p2 p3 p4 p5 p 6

pT

Mel-Cepstrumc1 c2 c3 c 4 c 5 c 6cT

MLSA Filter
SYNTHETIC SPEECH

Figure 3. Synthesis part of the system.

 fpreceding, current, succeedingg part-of-speech position of current phoneme in current accentualphrase fpreceding, current, succeedingg phoneme
Note that a context dependent HMM corresponds to aphoneme.
3.2. Decision-Tree Based Context Clustering
When we construct context dependent models taking account of many combinations of the above contextual factors,
we expect to be able to obtain appropriate models. However, as contextual factors increase, their combinations also
increase exponentially. Therefore, model parameters with
sufficient accuracy cannot be estimated with limited training data. Furthermore, it is impossible to prepare speech
database which includes all combinations of contextual factors.
To overcome this problem, we apply a decision-tree
based context clustering technique [11] to distributions forspectrum, pitch and state duration. The decision-tree based
context clustering algorithm have been extended for MSDHMMs in [13]. Since each of spectrum, pitch and duration
have its own influential contextual factors, the distributions for spectral parameter and pitch parameter and the state
duration are clustered independently (Fig. 2).

C_voiced
yesnoC_voiced

C_jaw_wide_vowel
yesnoR_silence
yes

yes
no

C_u
yes

no

R_silencenoyes

L_mouthnoC_silenceyesnoL_voicednoR_silence

no

C_N

yes

C_glottis
yesnoC_breath_position <= 1
yesL_breath_length <= 6noR_voicedyes

accent_type-mora <= 0noR_silenceyes

C_r"pit_s2_2"

yes
no

R_i
yes

R_back_vowel

noC_mouthno"pit_s2_1"

C_stop
yes

C_I

no

C_affricate(a) Tree for Spectrum Model(1st state)"dur_s2_187"
L_jaw_half_vowel

C_unvoiced_vowel
yes

R_breath_length <= 7
R_accent_length <= 8

C_U
nosentence_length <= 31
L_breath_length <= 5

C_N
yes

C_fricativenono

C_part-of-speech = 17

C_o
yes

C_pauseyes

C_voiced
R_silence(b) Tree for Pitch Model(1st state)(c) Tree for State Duration ModelFigure 4. Examples of decision trees.
4. TEXT-TO-SPEECH SYNTHESIS SYSTEM
The synthesis part of the HMM-based text-to-speech synthesis system is shown in Fig. 3. In the synthesis part, anarbitrarily given text to be synthesized is converted to acontext-based label sequence. Then, according to the label
sequence, a sentence HMM is constructed by concatenating
context dependent HMMs. State durations of the sentence
HMM are determined so as to maximize the likelihood of
the state duration densities [10]. According to the obtained
state durations, a sequence of mel-cepstral coefficients andpitch values including voiced/unvoiced decisions is generated from the sentence HMM by using the speech parameter
generation algorithm [4]. Finally, speech is synthesized directly from the generated mel-cepstral coefficients and pitch
values by the MLSA filter [5], [6].
5. EXPERIMENT
We used phonetically balanced 450 sentences from ATR
Japanese speech database for training. Speech signals weresampled at 16 kHz and windowed by a 25-ms Blackman
window with a 5-ms shift, and then mel-cepstral coefficientswere obtained by the mel-cepstral analysis3 . Feature vector
consists of spectral and pitch parameter vectors. Spectral
parameter vector consists of 25 mel-cepstral coefficients
including the zeroth coefficient, their delta and delta-deltacoefficients. Pitch parameter vector consists of log pitch, itsdelta and delta-delta. We used 3-state left-to-right HMMs
with single diagonal Gaussian output distributions. Decision trees for spectrum, pitch and duration models wereconstructed as shown in Fig. 2. The resultant trees for spectrum models, pitch models and state duration models had
6,615, 1,877 and 201 leaves in total, respectively.
3 The source codes of the mel-cepstral analysis/synthesis can be foundin http://kt lab.ics.nitech.ac.jp/˜tokuda/SPTK/ .

Fig. 4 shows examples of constructed decision trees forspectrum (a), pitch (b) and state duration (c). In these figures, “L ”, “C ” and “R ” represent “preceding”, “current” and “succeeding”, respectively. “Silence” representssilence of head or tail of a sentence, or pause. Questions of breath group and accentual phrase are represented by“ breath ” and “ accent ”, respectively. “Pit s2 ” and“dur s2 ” represent leaf nodes. From these figures, it isseen that spectrum models are much affected by phonetic
identity, pitch models for “voiced” are much affected byaccentual phrase and part-of-speech, and pitch models for“unvoiced” are clustered by a very simple tree. With regard
to state duration models, it can be seen that silence andpause models are much affected by accentual phrase andpart-of-speech, and the other models are much affected byphonetic identity.
Fig. 5, 6 show generated spectra and pitch pattern, respectively, for a Japanese sentence “heikiNbairituwo sageta keisekiga aru” which is not included in the training data.
Only the part corresponding to the first phrase “heikiNbairitu” is shown in Fig. 5.
The sound files4 attached to this paper demonstrate
our speech synthesis system. [Sound Y018S01.WAV]
and [Sound Y018S02.WAV] are vocoded natural speech.
[Sound Y018S11.WAV] and [Sound Y018S12.WAV] arespeech generated from the system, and correspond to[Sound Y018S01.WAV] and [Sound Y018S02.WAV], respectively. It is observed that the system synthesizes
natural-sounding speech which resembles the speaker inthe training database.
Through informal listening tests, we have found that thestopping rule (a minimum frame occupancy at each leaf
and a minimum gain in likelihood per splice) should be
4 These sound files were up-sampled to 22 kHz, and converted to WAV format.
The latest sound files can be found inhttp://kt lab.ics.nitech.ac.jp/˜yossie/TTS/.

Frequency (kHz) Frequency (kHz)0246802468Frequency

Figure 5. Generated spectra for a phrase “heikiNbairitsu”(top: natural spectra, bottom: generated spectra).
REFERENCES

200150100[1] T. Masuko, K. Tokuda, T. Kobayashi and S. Imai, “Speech synthesis
from HMMs using dynamic features,” Proc. of ICASSP, pp.389–392,
1996.

50

Frequency

01

2time (s)200150100[2] R. E. Donovan and E. M. Eide, “The IBM Trainable Speech Synthesis
System,” Proc. of ICSLP, vol.5, pp.1703–1706, 1998.
[3] M. Plumpe, A. Acero, H. Hon and X. Huang, “HMM-based Smoothing for Concatenative Speech Synthesis,” Proc. of ICSLP, vol.6,pp.2751–2754, 1998.

50012time (s)Figure 6. Generated pitch pattern for a sentence“heikiNbairitsuwo sageta keisekiga aru”
(top: natural pitch, bottom: generated pitch).
determined appropriately in decision tree construction. An
overly large tree will be overspecialized to training data andgeneralize poorly. On the other hand, a overly small tree
will model the data badly. Therefore we should introduce
some stopping criterion or cross-validation method (e.g.,[14]–[16]).
6. CONCLUSION
In this paper, we discribed an HMM-based speech synthesis system, in which spectrum, pitch and state duration aremodeled simultaneously in a unified framework of HMM.
As a result, it might be possible to synthesize speech withvarious voice characteristics, e.g., emotion expression, byapplying speaker adaptation or speaker interpolation technique.
Future work will be directed towards investigation ofcontextual factors and conditions of the context clustering,and evaluation of synthetic speech. Synthesizing speechwith various voice characteristics by applying speaker adaptation and speaker interpolation techniques is also our futurework.
ACKNOWLEDGEMENT
This work was partially supported by the Ministry of Education, Science, Sports and Culture Japan, Grant-in-Aid for
Scientific Research (B) 2, 1055125, 1998, Encouragementof Young Scientists, 0780226, 1998.

[4] K. Tokuda, T. Masuko, T. Yamada, T. Kobayashi and S.Imai, “An Algorithm for Speech Parameter Generation from Continuous Mixture
HMMs with Dynamic Features,” Proc. of EUROSPEECH, pp.757–760, 1995.
[5] T. Fukada, K. Tokuda, T. Kobayashi and S. Imai, “An adaptivealgorithm for mel-cepstral analysis of speech,” Proc. of ICASSP,
vol.1, pp.137–140, 1992.
[6] S. Imai, “Cepstral analysis synthesis on the mel frequency scale,”Proc. of ICASSP, pp.93–96, 1983.
[7] M. Tamura, T. Masuko, K. Tokuda and T. Kobayashi, “Speaker
Adaptation for HMM-based Speech Synthesis System Using MLLR,” Proc. of The Third ESCA/COCOSDA workshop on Speech
Synthesis, pp.273–276, 1998.
[8] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi and T. Kitamura,“Speaker Interpolation in HMM-Based Speech Synthesis System,”
Proc. of EUROSPEECH, vol.5, pp.2523–2526, 1997.
[9] K. Tokuda, T. Masuko, N. Miyazaki and T. Kobayashi, “HiddenMarkov Models Based on Multi-Space Probability Distribution for
Pitch Pattern Modeling,” Proc. of ICASSP, 1999.
[10] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi and T. Kitamura, “Duration Modeling in HMM-based Speech Synthesis System,”
Proc. of ICSLP, vol.2, pp.29–32, 1998.
[11] J. J. Odell, “The Use of Context in Large Vocabulary Speech Recognition,” PhD dissertation, Cambridge University, 1995.
[12] S. E. Levinson, “Continuously Variable Duration Hidden Markov
Models for Speech Analysis,” Proc. of ICASSP, pp.1241–1244,1986.
[13] N. Miyazaki, K. Tokuda, T. Masuko and T. Kobayashi, “A Study on
Pitch Pattern Generation using HMMs Based on Multi-space Probability Distributions,” Technical Report of IEICE, SP98-12, 1998 (inJapanese).
[14] H. J. Nock, M. J. F. Gales and S. J. Young, “A Comparative Studyof Methods for Phonetic Decision-Tree State Clustering,” Proc. ofEUROSPEECH, pp.111–115, 1997.
[15] K. Shinoda and T. Watanabe, “Speaker Adaptation with Autonomous
Model Complexity Control by MDL Principle,” Proc. of ICASSP,
pp.717–720, 1996.
[16] W. Chou and W. Reichl, “Decision Tree State Tying Based on Penalized Bayesain Information Criterion,” Proc. of ICASSP, pp.345–348,
1999.