ISCA Archive
http://www.isca-speech.org/archive

INTERSPEECH 2012
ISCA's 13th Annual Conference
Portland, OR, USA
September 9-13, 2012

An Evaluation of Parameter Generation Methods with Rich Context Models
in HMM-Based Speech Synthesis
Shinnosuke Takamichi1 , Tomoki Toda1 , Yoshinori Shiga2 ,Hisashi Kawai2 , Sakriani Sakti1 , Satoshi Nakamura1
1

Graduate School of Information Science, Nara Institute of Science and Technology,
8916-5 Takayama-cho, Ikoma, Nara 630-0192, Japan
2
National Institute of Information and Communications Technology, Japanshinnosuke-t@is.naist.jp, tomoki@is.naist.jp

Abstract
In this paper, we propose parameter generation methods using rich context models in HMM-based speech synthesis as yet
another hybrid method combining HMM-based speech synthesis and unit selection synthesis. In the traditional HMM-based
speech synthesis, generated speech parameters tend to be excessively smoothed and they cause muffled sounds in synthetic
speech. To alleviate this problem, several hybrid methods havebeen proposed. Although they significantly improve quality ofsynthetic speech by directly using natural waveform segments,
they usually lose flexibility in converting synthetic voice characteristics. In the proposed methods, rich context models representing individual acoustic parameter segments are reformed
as GMMs and a speech parameter sequence is generated fromthem using the parameter generation algorithm based on the
maximum likelihood criterion. Since a basic framework of the
proposed methods is still the same as the traditional framework,
the capability of flexibly modeling acoustic features remains.
We conduct several experimental evaluations of the proposed
methods from various perspectives. The experimental results
demonstrate that the proposed methods yield significant improves in quality of synthetic speech.
Index Terms: HMM-based speech synthesis, over-smoothing,rich context model, parameter generation

1. Introduction
Many attempts at developing a technique for converting text
into speech, i.e., Text-To-Speech (TTS), have been studied
for several decades. It is no doubtful that the corpus-based
approach [1] has yielded dramatic improvements of TTS. In
the corpus-based approach, there are two main synthesis techniques, i.e., sample-based synthesis and statistical parametricsynthesis. The sample-based synthesis such as unit selection [2]directly uses acoustic inventories selected from a speech corpus
for synthesizing a speech waveform. One of the main advantages of the unit selection is that high quality speech keeping
original voice characteristics is synthesized by concatenating
natural acoustic segments. However, characteristics of the generated speech are fully dependent on original voices.
On the other hand, the statistical parametric synthesis methods, such as HMM-based speech synthesis, use averaged acoustic inventories extracted from the speech corpus. In HMMbased speech synthesis, spectrum, pitch, and duration are modeled simultaneously in a unified framework of HMMs. In
synthesis these parameters are generated from HMMs undera maximum likelihood (ML) criterion by using dynamic features [3]. One of the biggest advantages of this method is

INTERSPEECH 2012

the capability of flexibly controlling voice characteristics, e.g.,
speaker-individuality control [4][5], and speaking-style control[6]. However, the generated speech parameters tend to be oversmoothed, and synthetic speech evidently sounds muffled compared with natural speech because detailed characteristics ofspeech parameters are often removed in the statistical process.
In order to alleviate this over-smoothing effect, hybrid
methods between those two main methods have been proposed.
Suitable waveform segments are searched out from the speech
corpus to maximize the HMM likelihood [7]. The use of waveform segments dramatically improves speech quality. However, it loses a strong advantage of HMM-based speech synthesis of controlling voice characteristics. As one of the hybrid approaches having better flexibility than unit selection, theuse of rich context models to represent each waveform segmentwith probability distributions of individual speech component
parameters, such as spectrum and F0 , has been proposed [9]. In
synthesis part, the probability distributions of all components
corresponding to one waveform segment are selected in each
HMM-state and speech parameters are generated from them.
This method also yields significant improvements in speech
quality. However, efficient and flexible acoustic modeling in theoriginal HMM-based speech synthesis is lost since this methodneeds to use a strong constraint among different components inthe selection of their probability distributions.
In this paper, we propose parameter generation methods foranother hybrid approach using the rich context models to keep
flexible property in synthesis part. The trained rich context
models are reformed as a Gaussian mixture model (GMM) ineach HMM-state. A speech parameter trajectory in each component is separately generated from the corresponding GMMs
using the ML criterion. The proposed methods also enables
to effectively use probability distributions of individual components from different waveform segments as in the original
HMM-based speech synthesis. We conducted several experimental evaluations from various perspectives to demonstrate theeffectiveness of the proposed parameter methods.
This paper is organized as follows. In section 2, traditional
HMM-based speech synthesis framework is briefly reviewed.
In section 3, the rich context modeling is described. In section
4, the proposed parameter generation methods are described. In
section 5, the experimental evaluation results and some discussions are given. Section 6 presents conclusions of this paper.

2. Traditional Framework
In HMM-based speech synthesis, various contextual factors areused to capture both segmental and prosodic features. Since1139

combinations of the contextual factors increase exponentiallyand the number of them is enormously large, one context label(called ”full context”) usually corresponds to only one acoustic
segment in training data. To robustly train context-dependent
HMMs, different full context labels are tied together in a decision tree structure and an output probability density function bc
is calculated in each leaf node of the decision tree in training
part [10], which is given bybc (ot ) = N (ot ; µc , Σc ) ,
(1)
[ ⊤
]
⊤
⊤
where ot = ct , ∆c⊤
is a feature vector includt , ∆∆ct
ing static feature, ct , and dynamic features, ∆ct ，∆∆ct . The
Gaussian distribution is denoted as N (·; µc , Σc ), where µcand Σc are a mean vector and a covariance matrix in the c-thleaf node, respectively.
In synthesis part, full context labels to be synthesized areclustered with the decision tree and output probability density
functions at corresponding leaf nodes are selected to form a sentence HMM. Then, a time sequence of the static feature vectors[]⊤ ⊤c = c⊤
is generated by maximizing the HMM
1 , · · · , cT
likelihood under a constraint on the explicit relationship between static and dynamic features (o = W c) [8]. It is wellknown that the generated speech parameters are over-smoothed
and this over-smoothing effect causes significant degradation insynthetic speech quality.

3. Rich Context ModelingIn the traditional approach, a single Gaussian distribution isused to model multiple acoustic segments belonging to the sameleaf nodes in the decision tree. Consequently its mean vector isexcessively smoothed and it becomes one of the factors causing the over-smoothing effect. On the other hand, the use ofmultiple acoustic segments is essential to robustly estimate its
covariance matrix. To alleviate the over-smoothing effect while
keeping robustness of parameter estimation high enough, in therich context model a mean vector is trained for each full context
label and a covariance matrix is tied over different full context
labels belonging to each leaf node of the decision tree [9]. The
output probability density function for the m-th full context label in the c-th leaf node is given by()
bc,m (ot ) = N ot ; µc,m , Σc .
(2)
The total number of different mean vectors is equivalent to thenumber of full context labels in training data. The total numberof different covariance matrices is equivalent to the number ofleaf nodes in the decision tree.
In training part, the context-clustered probability density
parameters are trained in a traditional way. Then, they are untied and only their mean vectors are further updated in every fullcontext label using forward-backward algorithm while tying thecovariance matrices over full context labels in each leaf node. In
synthesis part, probability density functions for all components
are jointly selected based on KL divergence between traditional
context-clustered model and the rich context model while avoiding selecting the probability density functions for individualcomponents from different acoustic segments. This process isregarded as unit selection but each acoustic segment is represented by probability density functions in individual components. Finally, speech parameter trajectories are generated fromthe selected probability density functions.

4. Parameter Generation Method Using
Rich Context Models

We propose parameter generation methods for selecting the richcontext models based on the ML criterion. The proposed meth-

INTERSPEECH 2012

Figure 1: Training and synthesis processes in proposed methods
ods keep flexibility in acoustic modeling of different speechcomponent parameters in the traditional framework, which islost in the conventional synthesis method described in section
3. The training and synthesis processes are shown in Figure 1.
4.1. Representation as GMM
After training the rich context models in the same manner asin the conventional method, the output probability density ineach leaf node is modeled by a GMM developed using all richcontext models in the same leaf node as follows:Mc∑()ωm N ot ; µc,m , Σc ,(3)
bc (ot ) =m=1

where ωm is the mixture component weight of the m-th richcontext model, and the total number of mixture components is
Mc . We can calculate the ML estimate of ωm based on theoccupancy counts given by forward-backward algorithm but inthis paper we set it to an equivalent value (ωm = 1/Mc ) overdifferent mixture components since we have found this weight
setting yields slight quality improvements in synthetic speech.
4.2. Parameter Generation
Given a state sequence q = [q1 , · · · , qT ]⊤ , which is determinedin a traditional way, the HMM likelihood is written as∑
P (o|q, λ) =P (o, m|q, λ),
(4)
all m
[
]
⊤ ⊤
where m = [m1 , · · · , mT ]⊤ , o = o⊤, and λ1 , · · · , oT
are a mixture component sequence (i.e., a rich context model
sequence), a feature vector sequence, and an HMM parameter
set, respectively. The static feature vector sequence is determined by maximizing the HMM likelihood under the constraint(o = W c) as in the traditional parameter generation process[8] as follows:∑ĉ = argmax
P (o, m|q, λ).
(5)
c
all m
4.2.1. EM algorithm
The ML estimate of c is determined with expectationmaximization (EM) algorithm. First, an initial static feature
vector sequence c(0) is determined. Then, the following auxiliary function isiteratively updating the posterior( maximized by )probability P m|W c(i) , q, λ given a current estimate c(i) in
E-step and a new estimate ĉ(i+1) while fixing it constant in

1140

(i)

Q c

(i+1)

,c
∑

93

)
=

92

(
)
(
)
P m|W c(i) , q, λ ln P W c(i+1) , m|q, λ .(6)all m

4.2.2. Approximation with a single mixture component sequence
We approximate the HMM likelihood given in Eq. (4) with asingle mixture component sequence as follows:∑
P (o, m|q, λ) ≃ P (o, m|q, λ) .
(7)
all m
After determining the initial static feature vector sequence c(0) ,the single mixture component sequence and the static feature
vector sequence is iteratively updated as follows :()
m̂(i+1) = argmax P m|W c(i) , q, λ ,
(8)
m
(
)
ĉ(i+1) = argmax P W c|m̂(i+1) , q, λ .
(9)c4.2.3. Discussion
One rich context model usually corresponds to one HMMstate acoustic segment. Therefore, the proposed processes arestrongly related to unit selection. In the proposed methods, theHMM likelihood for the static features and that for the dynamic
features are regarded as a target cost and a concatenation cost,
respectively [11]. The synthesis process with EM algorithm issimilar to the process of selecting multiple acoustic segments
and mixing them to generate speech parameters [12]. On theother hand, the synthesis process with a single mixture component sequence is similar to the process of selecting a single
acoustic segment sequence to generate speech parameters [2].
The proposed methods don’t have to use the constraint usedin the conventional selection method of the rich context models.
Therefore, the proposed methods keep an advantage of flexible
acoustic modeling in the traditional HMM-based speech synthesis. For instance, it is possible to separately search the bestrich context model sequences for different speech component
parameters to more widely cover a joint acoustic space. It isalso straightforward to apply different speech parameter generation methods to individual speech component parameters.
In the proposed methods, the rich context models are selected frame by frame. We can also select them state by state byusing an additional constraint that the same rich context model
is selected within the same HMM-state. Moreover, there aresome ways to determine the initial feature vector sequence. Oneof the reasonable ways is to use the sequence generated by thecontext-clustered HMMs in the traditional manner.

5. Experimental Evaluations
5.1. Experimental Conditions
We trained the context-dependent HSMM for a Japanese female speaker in a standard manner. We used 450 sentences
for phonetically balanced 503 sentences from ATR Japanese
speech database [13] for training, and 53 sentences for evaluation. Speech signals were sampled at 16 kHz. Mel-cepstral coefficients were extracted by STRAIGHT [14]. The shift length
was set to 5 ms. The feature vector consisted of 25 mel-cepstralcoefficients including the zeroth coefficient as a spectral parameter and both log-scaled F0 and 5 band-aperiodicity as excitation parameters. We used 5-state left-to-right HSMMs. The richcontext models were trained for only spectral component in thispaper. In synthesis, a global variance [15] was not considered .

INTERSPEECH 2012

63
Before iteration
After iteration

61

Before iteration
After iteration

59
91

57

Log Likelihood(

Log Likelihood

M-step :

90

89

55
53

51
49

88
47
87

45

Initial Feature

Initial Feature(a) Likelihood for generated pa- (b) Likelihood for natural paramrameter
eter

Figure 2: Effect of initial parameter sequence
We conducted two kinds of experimental evaluations. First,
we investigated the proposed methods from various perspectives. To clarify the effectiveness of the proposed synthesis
process in the spectral component, natural state duration determined by the state-level forced alignment with the conventional context-clustered models was used in the first evaluation.
In the other evaluation, the proposed method was evaluated infully synthesis process including duration, F0 , 5-band aperiodicity, and spectral parameter generation. Note that the conventional context-clustered models were used for duration, F0 , and5-band aperiodicity in all evaluations.
5.2. Evaluations in Natural State Duration
5.2.1. Effect of Initial Parameter Sequence
To investigate the effect of the initial parameter sequence on
the finally generated speech parameter sequence, we evaluatedthree settings of the initial parameter sequence; 1) Randomized:
generated from rich context models randomly selected in individual leaf nodes, 2) Conventional: generated from the conventional context-clustered models, and 3) Target: generated fromrich context models selected by maximizing the likelihood fornatural target speech parameters. We used the approximation ofa single mixture component sequence in the proposed method.
An initially selected rich context model sequence and a finallyselected rich context model sequence were evaluated with not
only the likelihood for the generated speech parameters but alsothat for natural speech parameters.
The result is shown in Figure 2. It is reasonable thatthe likelihood for the generated speech parameters increases
through the iteration as shown in Figure 2(a). On the other hand,
the likelihood for the natural speech parameters does not always
increase through the iteration and its value strongly depends on
the initial parameter sequence as shown in Figure 2(b). We canalso see that the likelihood differences in Figure 2(b) is muchlarger than those in Figure 2(a). From these results, it is shownthat the setting of the initial parameter sequence is essential andit is difficult to find the best rich context models using the likelihood measure.
5.2.2. Comparison of Proposed Parameter Generation Methods
To evaluate proposed methods, we compared synthetic speech
generated by the conventional clustered model (Conventional),
the proposed method with EM algorithm (Proposed(GMM)),
the proposed method with a single mixture component sequence(Proposed(single)), and the single mixture component sequence1141

find that the difference between ”Proposed(single)” and ”Proposed(target)” is larger in Figure 3(c) than that in Figure 3(a).
We will investigate what causes this difference in future work.

6. Conclusions
In this paper, we have proposed parameter generation methods
using rich context models in HMM-based speech synthesis toimprove quality of synthetic speech while keeping capability offlexibly modeling acoustic features. The rich context models
are reformed as the GMM and the likelihood measure is usedin speech parameter generation. Various experimental results
have demonstrated the effectiveness of the proposed methods.
However, it has also shown that suitable rich context models
are not selected by using the likelihood as a selection measure.
As future work, we will improve the selection measure.
Figure 3: Preference Scores for Speech Quality
selected by the natural speech parameters as a reference (Proposed(target)). The initial parameter sequence in the proposed
methods was generated by ”Conventional”. A preference test
(AB test) on speech quality was conducted. Every pair of thesefour types of synthetic speech was presented to seven listeners
in random order. Listeners were asked which sample sounds
better in terms of speech quality.
The result is shown in Figure 3(a). The proposed methods significantly improve speech quality. Moreover, the useof a single mixture component sequence yields better speech
quality than the use of EM algorithm. We can also see that thebest rich context model sequence, which is well approximatedwith ”Proposed(target)”, is difficult to select using the likelihood measure. This result is consistent with the result shown in
Figure 2(b).
5.2.3. Comparison of Selection Unit
To investigate the effect of the selection unit in the proposed
method, i.e., frame-based selection or state-based selection, we
compared synthetic speech generated by the conventional clustered model (Conventional), the proposed method with a single
mixture component sequence selected frame by frame (Framebased), and that selected state by state (State-based). In theselection process of the proposed method, the natural speechsamples were used as the target. We confirmed that the mixture component sequences selected by these two methods weredifferent from each other.
The result is shown in Figure 3(b). We can see that there isno significant difference between the frame-based selection andthe state-based selection and the state-based selection is also
effective for improving synthetic speech quality.
5.3. Evaluation in Full Synthesis
To confirm the effectiveness of the proposed method in fully
synthesis process where all speech parameters were generated
from the models, we compared synthetic speech generated by
the conventional clustered model (Conventional), the proposed
method with a single mixture component sequence selectedstate by state (Proposed(single)), and the single mixture component sequence selected state by state using the natural target
speech parameters as a reference (Proposed(target)). The initial
parameter sequence in the proposed method was generated by”Conventional”. A preference test (AB test) on speech quality
was conducted in the same manner as in Section 5.2.2.
The result is shown in Figure 3(c). It is observed thatthe proposed method yields significant quality improvements
in synthetic speech even in fully synthesis process. We can

INTERSPEECH 2012

Acknowledgements: Part of this research was executed under the Commissioned Research for ”Research and Development on Medical Communication Support System for Asian Languages based on Knowledge
and Language Grid” of National Institute of Information and Communications Technology (NICT), Japan.

7. References[1] Y. Sagisaka, ”Speech synthesis by rule using an optimal selectionof non-uniform synthesis units,” Proc.ICASSP, pp.679–682, 1988.
[2] N. Iwahashi, N. Kaiki, Y. Sagisaka, ”Speech segment selection
for concatenative synthesis based on spectral distortion minimization,” IEICE Trans, Fundamentals, Vol.E76-A, No.11, pp.1942–
1948, 1993.
[3] H. Zen, K. Tokuda, A. Black, ”Statistical parametric speech synthesis,” Speech Commun., Vol.51, No.11, pp.1039–1064, 2009.
[4] T. Yoshimura, T. Masuko, K. Tokuda, T. Kobayashi, T. Kitamura,”Speaker interpolation for HMM-based speech synthesis system”,
J. Acoust. Soc. Jpn. (E), Vol.21, No.4, pp.199–206, 2000.
[5] J. Yamagishi, T. Kobayashi, ”Average-voice-based speech synthesis using HSMM-based speaker adaptation and adaptive training,”
IEICE Trans. on Inf. and Syst., Vol.E90-D, No.2, pp.533–543,
2007.
[6] T. Nose, J. Yamagishi, T. Masuko, T. Kobayashi, ”A style control
technique for HMM-based expressive speech synthesis,” IEICE
Trans. Inf. and Syst., Vol.E90-D, No.9, pp.1406–1413, 2007.
[7] Z. Ling, L. Qin, H. Lu, Y. Gao, L. Dai, R. Wang, Y. Jiang, Z.
Zhao, J. Yang, J. Chen, G. Hu, ”The USTC and iflytek speechsynthesis systems for Blizzard Challenge 2007,” Proc. of Blizzard
Challenge workshop, 2007.
[8] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, T. Kitamura,”Speech parameter generation algorithms for HMM-based speechsynthesis,” Proc.ICASSP, pp.1315–1318, 2000.
[9] Z. Yan, Q. Yao, S.K. Frank, ”Rich Context Modeling for High
Quality HMM-Based TTS,” INTERSPEECH 2009, pp.1755–
1758, 2009.
[10] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, T. Kitamura, ”Simultaneous modeling of spectrum, pitch and duration in
HMM-based speech synthesis,” EUROSPEECH 1999, pp.2347–
2350, 1999.
[11] S. Kataoka, N. Mizutani, K. Tokuda, T. Kitamura, Decision treebacking-off in HMM-based speech synthesis, INTERSPEECH
2004, WeB1403p.12, 2004.
[12] T. Mizutani, T. Kagoshima, ”Concatenative Speech Synthesis
Based on the Plural Unit Selection and Fusion Method”, IEICE
Trans. on Inf. and Syst., Vol. E88-D, No.11, pp.2565–2572, 2005.
[13] Y. Sagisaka, K. Takeda, M. Abe, S. Katagiri, T. Umeda, H.
Kuwahara, ”A large-scale Japanese speech database,” ICSLP90,
pp.1089–1092, 1990.
[14] H. Kawahara, I. Masuda-Katsuse, A.D. Cheveigne, ”Restructuring speech representations using a pitch-adaptive time-frequency
smoothing and an instantaneous-frequency-based F0 extraction:
Possible role of a repetitive structure in souds,” Speech Commun.,Vol.27, No.3–4, pp.187–207, 1999.
[15] T. Toda, K. Tokuda, ”A speech parameter generation algorithm
considering global variance for HMM-based speech synthesis,”
IEICE Trans, Vol.E90-D, No.5, pp.816–824, 2007.

1142