This is the author’s version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/JSTSP.2013.2288599
JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012

1

Parameter Generation Methods
with Rich Context Models for High-Quality andFlexible Text-To-Speech Synthesis
Shinnosuke Takamichi, Nonmember, IEEE, Tomoki Toda, Member, IEEE, Yoshinori Shiga, Nonmember, IEEE,
Sakriani Sakti, Member, IEEE, Graham Neubig, Nonmember, IEEE, and Satoshi Nakamura Senior Member, IEEE.

Abstract—In this paper, we propose parameter generation
methods using rich context models as yet another hybrid method
combining Hidden Markov Model (HMM)-based speech synthesis
and unit selection synthesis. Traditional HMM-based speechsynthesis enables flexible modeling of acoustic features basedon a statistical approach. However, the speech parameters tend
to be excessively smoothed. To address this problem, several
hybrid methods combining HMM-based speech synthesis andunit selection synthesis have been proposed. Although theysignificantly improve quality of synthetic speech, they usuallylose flexibility of the original HMM-based speech synthesis. In
the proposed methods, we use rich context models, which arestatistical models that represent individual acoustic parameter
segments. In training, the rich context models are reformulated
as Gaussian Mixture Models (GMMs). In synthesis, initial speechparameters are generated from probability distributions overfitted to individual segments, and the speech parameter sequenceis iteratively generated from GMMs using a parameter generation method based on the maximum likelihood criterion. Since
the basic framework of the proposed methods is still the sameas the traditional framework, the capability of flexibly modelingacoustic features remains. The experimental results demonstrate:(1) the use of approximation with a single Gaussian component
sequence yields better synthetic speech quality than the use of
EM algorithm in the proposed parameter generation method,(2) the state-based model selection yields quality improvements
at the same level as the frame-based model selection, (3) the useof the initial parameters generated from the over-fitted speechprobability distributions is very effective to further improve
speech quality, and (4) the proposed methods for spectral andF0 components yields significant improvements in synthetic
speech quality compared with the traditional HMM-based speechsynthesis.
Index Terms—HMM-based speech synthesis, rich context
model, GMM, parameter generation, over-smoothing

I. I NTRODUCTION

T

EXT-To-Speech (TTS) is a technology that converts any
text into speech, and it plays an important role in many
speech applications. Many TTS techniques have been studied
for several decades. Recently, TTS systems are constructednearly automatically using pre-recorded speech. In general,
Copyright (c) 2013 IEEE. Personal use of this material is permitted.
However, permission to use this material for any other purposes must be
obtained from the IEEE by sending a request to pubs-permissions@ieee.org.
S. Takamichi, T Toda, S Sakti, G Neubig, and S Nakamura are with
the Graduate School of Information Science, Nara Institute of Science andTechnology, 8916–5 Takayama-cho, Ikoma, Nara, 630-0192, Japan.
Y. Shiga is with the National Institute of Information and Communications Technology, 3–5 Hikari–dai, Seika–cho, Soraku–gun, Kyoto, 619–0289,
Japan.

this type of TTS is called corpus-based TTS [1]. The corpusbased approach has yielded dramatic improvements of TTS as
researchers have been able to easily share common knowledge
and findings. In the corpus-based approach, there are two main
synthesis techniques: sample-based synthesis and statisticalparametric synthesis. Sample-based synthesis such as unit
selection [2], [3] directly uses acoustic inventories selectedfrom a speech corpus for synthesizing speech waveforms. As
shown in Fig. 1, one of the main advantages of unit selectionis the ability to synthesize that high-quality speech keeping
original voice characteristics by concatenating natural acoustic
segments [4]. However, characteristics of the generated speech
are fully dependent on original voices.
On the other hand, statistical parametric synthesis methods,
such as Hidden Markov Model (HMM)-based speech synthesis [5], use averaged acoustic inventories extracted fromthe speech corpus. In HMM-based speech synthesis, spectrum, pitch, and duration are modeled simultaneously in a
unified framework of HMMs. In synthesis these parameters
are generated from HMMs under the maximum likelihood(ML) criterion by using temporally dynamic features. Oneof the biggest advantages of this method is the capabilityof flexibly modeling and controlling acoustic features, e.g.,
speaker-individuality control [6], [7] and speaking-style control [8]. However, the generated speech parameters tend to beover-smoothed, and synthetic speech sounds muffled comparedwith natural speech because detailed characteristics of speechparameters are often smoothed out in the statistical process[9]. Consequently, quality of speech synthesized by HMMbased speech synthesis is still significantly lower than thatsynthesized by unit selection [10].
To address some problems of the sample-based synthesis method and the statistical parametric synthesis method,such as difficulties of automatically tuning cost functions forselecting waveform segments in unit selection synthesis or
the over-smoothing effect in HMM-based speech synthesis,
some hybrid methods combining these two methods have been
proposed [11], [12], [13], as also shown in Fig. 1. ML-based
unit selection synthesis [11] is one of the hybrid methods
to improve quality of synthetic speech. Suitable waveform
segments are searched out from the speech corpus to maximize
the likelihood of an HMM that is automatically trained using
a speech corpus. The use of waveform segments dramatically
improves speech quality compared with that in HMM-based
speech synthesis. However, it loses a strong advantage of

Copyright (c) 2014 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author’s version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/JSTSP.2013.2288599
JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012

Goal

Speech quality

ML-basedunit selection [11]Section II-C

2

TTS techniques are briefly reviewed. In Section III, the proposed parameter generation methods with rich context models
are described. In Section IV, the experimental evaluation
results are given. Section V presents conclusions.

Hybrid synthesis
Unit selection [2]Section II-A
Sample-based
synthesis

II. T EXT-T O -S PEECH T ECHNIQUESRich context
modeling [12]Section II-D

Proposed
Section III

HMM-based [5]Statistical
Section II-B
parametric synthesis

Flexibility

Fig. 1. Relationship diagram of the conventional methods and our proposed
method.

HMM-based speech synthesis, the ability to control voice characteristics. As one hybrid approach having better flexibility
than unit selection, rich context models that represent eachwaveform segment with probability distributions of individual
speech component parameters (spectrum and F0 ) have been
proposed [12]. In synthesis, the probability distributions of
all components corresponding to one waveform segment are
selected in each HMM-state and speech parameters are generated from them in the same manner as the original HMMbased speech synthesis. This method also yields significant
improvements in speech quality. However, the efficient andflexible acoustic modeling of the original HMM-based speechsynthesis is lost, as this method needs to use a strong constraint among different components in the selection of theirprobability distributions.
In this paper, we propose ML-based parameter generation
methods using rich context models as a hybrid method that
preserves the flexibility of the HMM-based speech synthesis.
The trained rich context models are reformulated as a Gaussian
Mixture Models (GMM) in each HMM state. The speechparameter trajectory in each component is separately anditeratively generated from the corresponding GMMs using theML criterion. The proposed methods also enable effective
use of probability distributions of individual components fromdifferent waveform segments as in the original HMM-based
speech synthesis. Therefore, they have better flexibility ofmodeling acoustic features compared with the other hybrid
methods as shown in Fig. 1. In the iterative generation process,a less-smoothed but highly discontinuous parameter sequenceis generated as an initial parameter sequence from probability
distributions over-fitted to individual segments, and then isiteratively refined by maximizing the HMM likelihood toachieve a less-smoothed and continuous parameter sequence.
We conduct several experimental evaluations of the proposed
methods applied to the spectral and F0 components. The
experimental results demonstrate that the proposed methods
yield significant improvements in quality of synthetic speech.
This paper is organized as follows. In Section II, several

A. Unit Selection Synthesis
Unit selection synthesis directly uses acoustic inventories
selected from a speech corpus for synthesizing a speech
waveform. In synthesis, an optimal set of acoustic segments isselected with the target information predicted by text analysis.
A target cost capturing the degradation of naturalness suchas caused by prosodic differences, and a concatenation costcapturing the degradation caused by concatenating acoustic
segments are often used as standard selection measures. The
optimal set is selected to minimize the cost function Cus
summarizing the target cost and the concatenation cost as
follows:
Cus = wtN∑

Ct (tn , un ) + wc

n=1

N
∑

Cc (un−1 , un ),

(1)

n=2

where tn and un are the n-th target and candidate acoustic
segments, respectively, Ct (tn , un ) and Cc (un−1 , un ) are thetarget cost function evaluating the difference between tn andun and the concatenation cost function evaluating the discontinuity at a joint point between un−1 and un , respectively,
wt and wc are the weights of the target and concatenation
cost functions, respectively. The weight of each cost function
is often determined manually on the basis of the result ofperceptual experiments [14].
One of the main advantages of unit selection synthesis isthat high-quality speech keeping the original voice characteristics is synthesized by concatenating natural acoustic segments.
However, voice characteristics of the generated speech are
fully dependent on the original voice. Therefore, it is difficult
to flexibly control voice characteristics.
B. HMM-based Speech Synthesis
Various contextual factors need to be considered to model
speech parameters in speech synthesis. Because combinationsof the contextual factors increase exponentially and the numberof them is enormous, one context label usually corresponds toonly one acoustic segment in training data. In HMM-based
speech synthesis, to robustly train context-dependent HMMs,
different full context labels are tied together in a decision tree[15]. In general, the decision tree for context clustering isconstructed based on the Minimum Description Length (MDL)
criterion [16], which is given by1∑
Γ (c) log |Σc | + aCD log Γ (0) ,2 c=1Cl(C) =(2)

where c is a leaf node index, C is the total number of leaf
nodes, a is a parameter to control C, D is the number offeature dimensions, Σc is a covariance matrix of leaf node c,and Γ (c) and Γ (0) are state occupancy counts in leaf node cand that in the root node, respectively.

Copyright (c) 2014 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author’s version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/JSTSP.2013.2288599
JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 20123

After the tree-based context clustering, the output probability density function bc (clustered model) is calculated foreach leaf node. Different decision trees are constructed forindividual speech components (spectrum, F0 , and duration)[15].
Spectral component: Spectral parameters are modeled bya continuous HMM. Its state output probability is given bybc (ot ) = N (ot ; µc , Σc ) ,(3)where µc and Σc are the mean vector and covariance matrixof leaf node c. The Gaussian distribution with a mean vectorµc and a covariance matrix Σc is denoted as N (·; µc , Σc ). In
HMM-based
speech synthesis,
a feature vector is defined as
[
]
⊤
⊤ ⊤
ot = c⊤, which includes static feature ct ,t , ∆ct , ∆∆ct
and dynamic features ∆ct ，∆∆ct . These dynamic features
are computed from ct by
(1)

L+

∆ct =

∑

ωτ(1) ct+τ ,

(4)

(1)
τ =−L−∆∆ct =

ωτ(2) ct+τ ,

(5)

(2)

τ =−L−
(n)

(n)

(n)

where ωτ , L− , and L+ are the n-th order weight coefficient and frame lengths for computing dynamic features.
F0 component: F0 parameters are modeled by a MultiSpace Distribution HMM (MSD-HMM) [17]. Its state output
probability is given by{
wc N (ot ; µc , Σc ) , lt = V
bc (ot ) =,(6)
1 − wc ,lt = U
where lt is a discrete voicing label that is either voiced V or
unvoiced U at frame t, and wc is the weight of the voiced
space of leaf node c, respectively. Note that lt is observable
together with ot .
As additional speech parameters, aperiodic parameters areoften used and are modeled with continuous HMMs.
In synthesis, full context labels to be synthesized areclustered with decision trees, and the output probability density functions at corresponding leaf nodes are selected toform a sentence HMM. After determining state durations⊤
q = [q1 , · · · , qT ] , a time sequence of static feature vectors
[ ⊤
]
⊤ ⊤
c = c1 , · · · , cT
is generated by maximizing the HMM
likelihood under a constraint on the relationship between staticand dynamic features (o = W c), as follows:ĉ

v (c) =
v (d) =⊤[v (1) , · · · , v (Dc )] ,T
1∑2
(ct (d) − c (d)) ,T t=1
1∑
ct (d),
T t=1

(9)
(10)

T

c (d) =

(11)

where Dc is the number of dimensions of the static feature
vectors. The GV likelihood P (v (c) |λv ) is modeled by asingle Gaussian distribution with the mean vector µv and thecovariance matrix Σv as follows:
P (v (c) |λv ) = N (v (c) ; µv , Σv )(12)

where λv is a parameter set of the GV. The GV likelihood
is estimated using natural speech parameter sequences. In
parameter generation methods considering the GV, a speechparameter sequence c is generated to maximize product of the
HMM likelihood and the GV likelihood, which is given by
w

(2)

L+

∑

[9]. GV is defined as the variance of the static feature vectors
over an utterance, which is calculated as:= argmax P (o|q̂, λ)c= argmax P (W c|q̂, λ) ,
c

(7)
(8)

where W is the weighting matrix for calculating the dynamic[18],] λ is the parameter set of the HMM, and o =[features
⊤
o⊤
,
·
·
·
,
o⊤
is a feature vector sequence.
1
T
One of the well-known approaches for quality improvements in synthetic speech is the use of Global Variance (GV)ĉ = argmax P (W c|q̂, λ) P (v (c) |λv ) GV ,(13)cwhere wGV is the weight of GV likelihood.
One of the biggest advantages of HMM-based speechsynthesis is the capability to flexibly control voice characteristics. However, the speech parameters generated by HMMbased speech synthesis are over-smoothed because the detailed
characteristics of speech parameters are often removed in thestatistical process. This over-smoothing effect, which causes
significant degradation in synthetic speech quality is alleviatedby considering the GV. However, it is known that this method
often causes artificial sounds, such as clicks, pops, and shorthigh-pitched whines. Therefore, another approach to alleviate
the over-smoothing effect without causing any artifacts isrequired.
C. Hybrid Synthesis with Waveform Concatenation
In order to avoid manually tuning cost functions used in unit
selection synthesis, ML-based unit selection synthesis [11] hasbeen proposed to combine unit selection synthesis and HMMbased speech synthesis. In training, after the standard HMMs
are trained in the same process as in HMM-based speechsynthesis, two additional statistical models called the phone
duration model and the concatenation model are trained. The
phone duration model represents the duration of each phonesized acoustic segment. On the other hand, the concatenation
model is defined as the differential of acoustic features between the first frame of the current phone and the last frameof the previous phone.
In synthesis, the optimal set of acoustic segments is selectedfrom the speech database to maximize the cost function combining the likelihoods of HMM, the phone duration model, andthe concatenation model. The cost function CML is represented
in the same form as that of unit selection, which is given by
CML =

N
∑
n=1

Ct (un ) +N∑

Cc (un−1 , un ),

(14)

n=2

Copyright (c) 2014 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author’s version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/JSTSP.2013.2288599
JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012

The use of acoustic segments dramatically improves speech
quality. Moreover, introducing an automatically determined
cost function is an effective way to avoid the manual setting ofeach cost function. However, the waveform generation processusing waveform concatenation loses the flexibility of HMMbased speech synthesis.

4and F0 components where rn represents the rich context modelin the n-th state
DKL (g, r) =DKL (gn , rn ) =N∑

DKL (gn , rn ) Tn ,
(17)
n=1
(SP)
(F0)
DKL (gn , rn ) + DKL (gn , rn ) , (18)(SP)

D. Hybrid Synthesis with Parameter Generation
As one of the hybrid approaches having better flexibility
than the standard unit selection synthesis or the ML-based
unit selection synthesis, the use of rich context models torepresent each acoustic segment with probability distributionsof individual speech component parameters, such as spectrum
and F0 has been proposed [12]. In the traditional HMMbased speech synthesis, a single Gaussian distribution is used
to model multiple acoustic segments belonging to the sameleaf nodes in the decision tree. Consequently its mean vector
is excessively smoothed and it becomes one of the factors
causing the over-smoothing effect. On the other hand, theuse of multiple acoustic segments is essential to robustly
estimate the model parameters, in particular its covariancematrix. Although the use of GMMs as each state output
probability distribution reduces the over-smoothing effect [18],its reduction effect is limited. To alleviate the over-smoothing
effect while preserving robustness of the parameter estimation,in rich context model a mean vector is trained for each fullcontext label and a covariance matrix is tied over different fullcontext labels belonging to each leaf node of the decision tree
[12].
Spectral component: The output probability density function bc,m of the rich context model for the m-th full context
label in the c-th leaf node is given by()
bc,m (ot ) = N ot ; µc,m , Σc .
(15)
F0 component: Same as the spectral parameters, a mean
vector of the Gaussian distribution in voiced space is updated
as follows:(){
wc N ot ; µc,m , Σclt = Vbc,m (ot ) =.
(16)
1 − wc ,lt = UThe total number of different mean vectors is equivalent to thenumber of full context labels in training data. The total numberof different covariance matrices is equivalent to the numberof leaf nodes in the decision tree. In training, parameters of
the clustered models are estimated in the traditional mannerof HMM-based speech synthesis. Then, they are untied andonly their mean vectors are further updated in every fullcontext label using the Baum-Welch algorithm while tying thecovariance matrices over full context labels in each leaf node.
In synthesis, full context labels to be synthesized areclustered with the decision trees and the clustered models
at corresponding leaf nodes are determined as a target g ={g1 , · · · , gN } where gn represents the clustered model in
the n-th state. Then, a sequence of the rich context models
r = {r1 , · · · , rN } is selected to minimize the following
Kullback-Leibler Divergence (KLD) considering both spectral

where DKL (·) is the total KLD, DKL (gn , rn ) and(F0)
DKL (gn , rn ) are KLDs for spectral and F0 components
respectively, and Tn is a state duration in the n-th state.
The rich context models for spectral and F0 components
are selected simultaneously using a constraint among different
components (spectrum and F0 ). This selection process canbe regarded as unit selection, where each acoustic unit isrepresented as a joint probability distribution of both spectrum
and F0 . This approach also yields significant improvementsin speech quality. However, efficient and flexible acoustic
modeling in the original HMM-based speech synthesis is lostby the use of the strong constraint in the model selection.
III. PARAMETER G ENERATION M ETHODS WITH R ICH
C ONTEXT M ODELS BASED ON THE ML C RITERION
A. Reformulation of GMM Using Rich Context Models
An overview of the proposed method is shown in Fig. 2.
In the proposed method, the rich context models are trainedfor each leaf node after training conventional clustered models.
In synthesis, after determining the leaf nodes corresponding tofull context labels to be synthesized, the rich context models
must be selected from a large number of models in the leaf
nodes. The proposed methods introduce a model selection
process based on the ML criterion. After training the richcontext models in the same manner as in the conventional
method described in Section II-D, the output probability
density in each leaf node is modeled using a GMM developedwith all rich context models in the same leaf node as follows:
Spectral component:bc (ot ) =

Mc
∑

(
)
ωm N ot ; µc,m , Σc ,

(19)

m=1

where ωm is the mixture component weight of the m-th richcontext model, and the total number of mixture components
is Mc . We can calculate the ML estimate of ωm based on theoccupancy counts given by the forward-backward algorithm
but in this paper we set it to an equivalent value (ωm = 1/Mc )over different mixture components since we have found thisweight setting yields slight quality improvements in synthetic
speech.
F0 component: The F0 component is calculated as

(
)
∑c
 M
wc,m N ot ; µc,m , Σclt = Vbc (ot ) =, (20)
 m=1
1 − wc ,lt = U
where ωc,m is the mixture component weight in voiced spaceof the m-th rich context model. We set it to an constant value(ωc,m = ωc /Mc ) based on our previous findings as mentioned
above.

Copyright (c) 2014 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author’s version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/JSTSP.2013.2288599
JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012

B. Parameter Generation Methods

5

Continuous HMM

MSD-HMM

Given a state sequence q̂ [15], which is determined in the
traditional HMM-based speech synthesis, the HMM likelihood
is written as∑
P (o|q̂, λ) =P (o, m|q̂, λ),
(21)
all m
where m = {m1 , · · · , mT } is a mixture component sequence.
The static feature vector sequence is determined by maximizing the HMM likelihood under the constraint (o = W c) as in
the traditional parameter generation process [18] as follows:∑ĉ = argmax
P (o, m|q̂, λ).
(22)
c
all m

Training: Space weight

Clustered model

Update mean vector
Rich context model
Reformulate
GMM
Approximation

EM

1) EM algorithm: The ML estimate of c is determinedwith the EM algorithm. First, an initial static feature vector
sequence c(0) is determined. Then, the following auxiliary
function is maximized
by iteratively
updating the posterior()
probability P m|W c(i) , q̂, λ given a current estimate c(i)in the E step and a new estimate ĉ(i+1) , while fixing it constantin the M step:()
Q c(i) , c(i+1) =
(
)
(
)
∑
P m|W c(i) , q̂, λ ln P W c(i+1) , m|q̂, λ .(23)all mThis equation is solved with the conventional generation
method using the HMM-GMM [18].
2) Approximation with Single Gaussian: We approximate
the HMM likelihood given in Eq. (21) with a single mixture
component sequence as follows:∑
P (o, m|q̂, λ) ≃ P (o, m̂|q̂, λ) . (24)P (o|q̂, λ) =all m
After determining the initial static feature vector sequencec(0) , the single mixture component sequence and the staticfeature vector sequence are iteratively updated as follows:()
m̂(i+1) = argmax P m|W ĉ(i) , q̂, λ ,
(25)
m
(
)
ĉ(i+1) = argmax P W c|m̂(i+1) , q̂, λ . (26)c
Eq. (26) is solved in the same manner as traditional HMMbased speech synthesis.
C. Initialization Method with Tree-based Context Clustering
In the proposed parameter generation methods, we need todetermine an initial parameter sequence. One of the straightforward ways is to use the parameter sequence generated by
the clustered models of the traditional HMM-based speechsynthesis. However, these initial parameters are not effective in
improving quality. As we will show in Section IV-D, speechparameters generated by the proposed methods are strongly
dependent on the setting of the initial parameters. Although thetransitions of this initial parameter sequence are continuous,
the parameter sequence is over-smoothed. Consequently, theSynthesis

Initial parameters
Model selection
Generated parameters

Fig. 2. An overview of the proposed generation methods.

Decision tree forgenerated parameters

Decision tree for
initial parameters

Clustered modelRich context model
GMM

Initial parameters

Parameter generation

Proposed

Generated parameters

Fig. 3. An overview of the proposed initialization method.

parameters finally generated through the iterative generation
process tends to still be over-smoothed.
To generate a less-smoothed initial parameter sequence,
we propose an initialization method with over-fitted models
generated by tree-based context clustering. As shown in Fig.
3, a large-sized tree for context clustering is constructed bydecreasing the parameter a of the MDL criterion shown in
Eq. (2). Note that the sufficient statistics to build this tree are
the same as those used in calculating rich context models,
which are extracted using the conventional clustered models.
Therefore, its tree structure is slightly different from thatof the decision tree for rich context models, which is thesame as that for the conventional clustered models and isbuilt using different sufficient statistics before developing theconventional clustered models.

Copyright (c) 2014 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author’s version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/JSTSP.2013.2288599
JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012

1st mel-cepstral coefficient

4
3.5
3
2.5
2
1.5
1
0.5
0
-0.5
400

6

Proposed initial parameters
Proposed generated parameters
Initial parameters by
conventional clustered models

450

500

550
600
Frame index

650

700

Fig. 4. An example of initial and generated mel-cepstral parameter sequence.

In this tree, the mean vector and covariance matrix of theprobability density function are calculated from only a few
acoustic inventories determined by context factors. Therefore,
compared to a parameter sequence generated by the conventional clustered model, a less-smoothed initial parametersequence is generated. It is expected that this initial parametersequence helps to select a less-smoothed model sequence inthe parameter generation process with rich context models.
On the other hand, the use of a larger-sized decision tree
causes over-fitting problems. In particular, the initial parametersequence significantly suffers from discontinuous transitions.
In the parameter generation with rich context models, thisproblem of discontinuity is addressed by the use of tiedcovariance matrices in the rich context models and the model
selection based on the likelihoods for both static and dynamic
features. An example of initial and generated parameter sequences by the proposed method is shown in Fig. 4. As acomparison, an initial parameter sequence generated by theconventional clustered models is also shown in this figure. We
can see that discontinuous transitions in the initial parametersequence are alleviated in the generated parameter sequence.
For the F0 contour generation, the voiced/unvoiced interval
is determined by that of the initial parameter sequence determined by the weight of the voiced space of the clusteredmodels in the larger-sized decision tree.
D. Discussions
One rich context model usually corresponds to one HMMstate acoustic segment. Therefore, the proposed processes arestrongly related to unit selection synthesis. In the proposed
method, the HMM likelihood for the static features and thatfor the dynamic features are regarded as a target cost and aconcatenation cost, respectively [19], [20].
The synthesis process with the EM algorithm is similar
to the process of selecting multiple acoustic segments andmixing them to generate speech parameters [21]. On the other
hand, the synthesis process with a single mixture component
sequence is similar to the process of selecting a single acoustic
segment sequence to generate speech parameters [2].
The proposed parameter generation methods don’t have to
use the constraint used in the conventional selection methodof the rich context model and still keep the acoustic modelingframework the same as that of traditional HMM-based speechsynthesis. Therefore, the proposed methods preserve the advantage of flexible acoustic modeling provided by traditional
HMM-based speech synthesis. For instance, it is possible toseparately search for the best rich context model sequences fordifferent speech component parameters to more widely cover a
joint acoustic space. Moreover, the probabilistic representation
with GMMs in the proposed method makes it possible tostraightforwardly use various techniques proposed for HMMGMMs, such as flexible control of the model complexity
according to individual speech components using traditional
information criteria to develop a scalable system, and model
adaptation directly using the conventional techniques, suchas maximum likelihood linear regression [22]. It also hasa potential to optimize the covariance matrices of the richcontext models are also optimized in the proposed methodby directly using the conventional methods, such as crossvalidation [23]. On the other hand, it is not straightforward
to apply these techniques to the framework based on theconventional selection method of the rich context models, andtherefore, some modifications would be required. It is alsostraightforward to apply different speech parameter generation
methods to individual speech component parameters such as
the conventional method with or without the GV [9].
In the proposed parameter generation methods, the richcontext models are selected frame by frame. We can also selectthem state by state by using an additional constraint that thesame rich context model is selected within the same HMM
state. In the state-based model selection for the F0 component,
the voiced/unvoiced region in each HMM state is determinedby the ratio of the number of voiced frames to that of unvoiced
frames.
IV. E XPERIMENTAL E VALUATION
A. Experimental Conditions
We trained a context-dependent phoneme Hidden SemiMarkov Model (HSMM) [24] for a Japanese female speaker.
We used 450 sentences for training and 53 sentences forevaluation from phonetically balanced 503 sentences including
in ATR Japanese speech database [25]. Speech signals were
sampled at 16 kHz. The shift length was set to 5 ms. The
0th-through-24th mel-cepstral coefficients were extracted as
spectral parameters and log-scaled F0 and 5 band-aperiodicity[26], [27] were extracted as excitation parameters by the
STRAIGHT (Speech Transformation and Representation using Adaptive Interpolation of weiGHTed spectrum) analysis
system [28]. The feature vector consisted of spectral andexcitation parameters and their delta and delta-delta features.
Five-state left-to-right HSMMs were used. In synthesis, global
variance (GV) [9] was not considered. Both conventional
clustered models and the rich context models were constructed.
Then, the rich context models were reformulated as GMMs
using the proposed methods. Table I shows the numbers of leaf
nodes in the conventional clustered models and the rich context
models. The average numbers of the mixture components
were 186.3 for the spectral component and 60.7 for the F0
component.
We conducted five kinds of experimental evaluation. In thefirst evaluation, we compared the two proposed parameter

Copyright (c) 2014 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author’s version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/JSTSP.2013.2288599
JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012

1
95% confidence interval

Preference score

generation methods described in Section III-B. In the second
evaluation, we compared frame-based and state-based model
selection to investigate the effect of the model selection unit.
In the third evaluation, we investigated the effects of the initial
parameter sequence on the generated parameter sequence. In
the fourth evaluation, we investigated the effectiveness of the
proposed initialization method described in Section III-C.
In the last evaluation, we applied proposed methods to both
spectral and F0 components to confirm the effectiveness ofproposed methods. Conventional clustered models were used
for duration and aperiodic components in all evaluations.

7

0.8
0.6
0.4
0.2
0

Conv

GMM

SingleProposed

TargetFig. 5. Preference score on speech quality for comparing two proposed
generation methods.

B. Comparison of Proposed Parameter Generation Methods

TABLE IN UMBERS OF LEAF NODES IN THE CONVENTIONAL CLUSTERED MODELSAND THE RICH CONTEXT MODELS .
Component

Spectrum

Model

Clustered models
Rich Context Models

F0

Clustered models
Rich Context Models

State
1
2
3
4
5
1-5
1
2
3
4
5
1-5

Number of leaf nodes
151
160
171
105
141
27118
497
473
580
374
310
27118

1

95% confidence interval

Preference score

1) Comparison of Proposed Methods: To evaluate two
proposed generation methods, we compared synthetic speech
generated by the conventional clustered model (Conv), theproposed generation method with EM algorithm (Proposed(GMM)), that with a single mixture component sequence (Proposed (Single)), and the single mixture component sequence
selected by the natural speech parameters as a reference (Target). The initial parameter sequence in the proposed generation
methods was generated by “Conv.” Note that the proposed
generation methods are applied to only the spectral component,
and the clustered model is used for the F0 component. A
preference test (AB test) on speech quality was conducted.
Every pair of these four types of synthetic speech was presented to seven listeners in random order. Listeners were asked
which sample sounds better in terms of speech quality. Notethat natural state duration determined by the state-level forcedalignment with the conventional context-clustered models wasused to clarify the effectiveness of the proposed methods in abetter setting.
The result is shown in Fig. 5. The proposed methods significantly improve speech quality. Moreover, the use of a single
mixture component sequence yields better speech quality thanthe use of the EM algorithm. We can also see that the best richcontext model sequence, which is approximated with “Target,”is difficult to select using the likelihood measure. In thefollowing experiments, the parameter generation method using
approximation with a single mixture component sequence wasused as the proposed parameter generation method.

0.8
0.6
0.4
0.2
0

Conv

Proposed (Clus)

TargetFig. 6. Preference scores on speech quality for investigating quality improvements under generated duration.

2) Evaluation in Generated Duration: To investigate theeffectiveness of the proposed generation method under thegenerated duration, we compared synthetic speech: 1) Conv:
generated from conventional clustered models, 2) Proposed(Clus): generated using the parameter sequence of “Conv” asthe initial parameters in the proposed generation method, 3)
Target: generated using natural target speech parameters as
the initial parameters in the proposed generation method. A
preference test (AB test) by seven listeners on speech quality
was conducted in the same manner as in Section IV-B1.
Note that the proposed method is applied to only spectral
parameters.
The result of the preference test is shown in Fig. 6. It
is observed that the proposed generation method yields only
slight improvements in synthetic speech. On the other hand,
we can find that the difference between “Proposed (Clus)” and“Target” is larger than that in Fig. 5. From this result, we canfind that the state duration affects the quality improvements inthe proposed generation method. Synthetic speech using thegenerated state duration sounds more muffled compared withthat using the natural state duration. It is expected that thisquality degradation is caused by the quality differences of theinitial parameter sequences because we have found that similar
quality differences between the generated state duration andthe natural state duration are also observed in synthetic speechusing the conventional clustered models.
Although we did not conducted similar experiments for theF0 component in this section, we believe that it will show thesame results in the F0 component.

Copyright (c) 2014 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author’s version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/JSTSP.2013.2288599
JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 20128

Normalized mixture
component index

1
0.8

Frame-based

0.6
0.4

State-based

0.2
0
190

210

Frame index

Fig. 7. An example of mixture component sequences selected by framebased and state-based model selection (the selected mixture component index
is normalized by the total number of mixture components.).

1

95% confidence interval

Preference score

0.8

Preference score

0.8
0.6

0.6

0.4

0.4

0.2
0

1

95% confidence interval

0.2

Conv

Frame State
Proposed0Conv(a) Spectrum

Frame State
Proposed(b) F0Fig. 8. Preference scores on speech quality for comparing the selection unit

C. Comparison of Model Selection Unit
To investigate the effect of the model selection unit in the
proposed generation method, i.e., frame-based selection orstate-based selection, we compared synthetic speech generated
by the conventional clustered model (Conv), the proposed
method with a single mixture component sequence selected
frame by frame (Proposed (Frame)), and that selected state
by state (Proposed (State)). The natural state duration wasused to investigate the effects in a better setting. The proposed generation method was applied to each spectral and F0
component, and the natural speech parameter sequence wasused as initial parameters. The conventional clustered models
were used for the component that the proposed methods were
not applied to. A preference tests by 7 listeners on speech
quality were conducted for spectral and F0 component in thesame manner as in the Section IV-B. We confirmed that the
mixture component sequences selected by these two methods
were different from each other as shown in Fig. 7.
The result for spectral component is shown in Fig. 8(a), andthat for the F0 component is shown in Fig. 8(b). We can seethat there is no significant difference between the frame-based
selection and the state-based selection in spectral components.

Moreover from Fig. 8(b), the same result is observed in theF0 component even though the U/V intervals are also different
from each other. From these results, the state-based selectionis also effective for improving synthetic speech quality inboth spectral and F0 components. We can also see that thedifference between “Conv” and “Proposed (Frame)” is larger
in the spectral component than that in the F0 component. This
result shows that the improvements yielded by the rich context
models for the spectral component is larger than those for theF0 component.
D. Investigation of Dependency on Initial Parameter Sequence
To investigate the dependency on the setting of initial
parameter sequence on the finally generated speech parametersequence after the proposed parameter generation, we evaluated three settings of the initial parameter sequence: 1) Rand:
generated from rich context models randomly selected fromindividual leaf nodes, 2) Clus: generated from the conventional context-clustered models, and 3) Target: natural target
speech parameters. The initially selected rich context model
sequence and the finally selected rich context model sequence
were evaluated with the model likelihood for the generated
speech parameters and that for natural speech parameters. Thisevaluation was conducted for each spectral and F0 components
under the natural state duration.
The result of HMM likelihood for the generated parameters
for the spectral components is shown in Fig. 9(a), that for
the natural parameters in the spectral components is shownin Fig. 9(b), and those for the F0 component are shown in
Fig. 10(a) and Fig. 10(b), respectively. Because the HMM
likelihood for the generated parameters is the criterion for theparameter generation, it is reasonable that the likelihood forthe generated speech parameters increases through iterationin both components as shown in Fig. 9(a) and Fig. 10(a).
On the other hand, the likelihood for the natural speechparameters does not always increase through iteration and its
value strongly depends on the initial parameter sequence asshown in Fig. 9(b) and Fig. 10(b). We can also see that thelikelihood differences in Fig. 9(b) and Fig. 10(b) are muchlarger than those in Fig. 9(a) and Fig. 10(a). These results
suggest that the proposed generation method strongly depends
on the initial parameter.
E. Effectiveness of Initialization Method
1) Confirmation of Alleviating Discontinuous Transition:
Before investigating the effectiveness of the proposed initialization method, we conducted a preliminary experiment
to confirm whether or not the proposed iterative parameter
generation method effectively alleviates the discontinuoustransition in the initial parameter sequence. We evaluated threesettings of the initial parameter sequences: 1) Clus: initial
parameters generated from the conventional clustered models,
2) a = 0.1: initial parameters generated with a large-sized
decision tree (a = 0.1), and 3) Target: natural target speechparameter sequence as a target reference. The difference of
HMM likelihoods for the generated parameters between theinitially selected rich context model sequence and the finally

Copyright (c) 2014 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author’s version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/JSTSP.2013.2288599
JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012

62
Before iteration
After iteration60

91

Likelihood difference

92

9Before iteration
After iteration

Log likelihood

Log likelihood

58
56

90

54

89

52

48
87

Rand

Clus

Target(a) For the generated parameters.

46

Rand

Clus

3.8
Before iteration
After iteration

Log likelihood

3.6
Log likelihood

3.89

3.4
3.2

3.88

3
3.87

Rand

Clus

Target(a) For the generated parameters.

2.8

For delta-delta feature vector21
a = 0.1

Clus

TargetTarget(b) For the natural speech parameters.

Before iteration
After iteration

For delta feature vector

Fig. 11. Differences of HMM likelihood between before and after iteration.

Fig. 9. HMM likelihood of selected rich context models (spectrum).

3.9

For static feature vector

3

0

50

88

4

Rand

Clus

Target(b) For the natural speech parameters.

Fig. 10. HMM likelihood of selected rich context models (F0 ).

selected rich context model sequence was calculated for eachstatic and dynamic features in the spectral parameters.
The result of the likelihood differences caused by theiterative parameter generation is shown in Fig. 11. It wasobserved that the HMM likelihood for dynamic features of“a = 0.1” increases more than that in the other initial
parameter sequences. From this result, we can see that thediscontinuous transitions in the initial parameter sequence are
alleviated by the iterative parameter generation.
2) Objective Evaluation of Initialization Method: To investigate the tree size used to generate the initial parametersequence, we evaluated 3 settings of the initial parameters:
1) Clus: initial parameters generated from the conventional
clustered models, 2) Proposed: initial parameters generated
with a large-sized decision tree (a = 0.1, 0.2, · · · , 1.0), and
3) Target: the natural target speech parameter sequence as atarget reference for each spectral and F0 components. The
rich context model sequences finally selected by the parameter
generation method using these initial parameter settings were
evaluated with the HMM likelihood for the natural speechparameters. The tree size was calculated as the ratio of thenumber of leaf nodes of the decision trees compared to
the number of full context models. Moreover, the parametersequences generated by the selected rich context models (i.e.,
those generated by the proposed parameter generation method)are evaluated with both the GV likelihood [9] and U/V error
rate. The U/V error rate for the F0 component is calculated
as the ratio of the number of U/V mismatched frames in
the generated parameter sequence compared to the naturalparameter sequence.
The result of HMM likelihood in the spectral component isshown in Fig. 12(a), that of GV likelihood in the spectral component is shown in Fig. 12(b), and those in the F0 component
are shown in Fig. 13(a) and Fig. 13(b), respectively. Moreover,the size of decision trees used in the proposed initialization
method is shown in Fig. 14, and the result of U/V error rateis shown in Fig. 15. It is observed from Fig. 12(a) that the
HMM likelihood of “Proposed” very slightly increases as theparameter a decreases from 1.0 to 0.5, and it rapidly decreases
as the parameter a decreases more in the spectral components.
We can see that the HMM likelihood at a = 0.5 is almost thesame as that of “Clus” but it is significantly lower than thatof “Target.” The result for the F0 component shown in Fig.
13(a) is similar to this result except that no peaks appeared as
the parameter a decreases. On the other hand, It is observed
from Fig. 12(b) that the GV likelihood of “Proposed” rapidly
increases as the parameter a decreases, and its value at a = 0.1
is higher than that of “Target” in the spectral component. In
the F0 component, the GV likelihood of “Proposed” rapidly
increases as the parameter a decreases from 1.0 to 0.6, andit rapidly decreases beyond 0.6. Moreover from Fig. 15, we
can see that the U/V error rate increases as the parameter adecreases.
3) Subjective Evaluation of Initialization Method: To confirm the effectiveness of the proposed initialization method,
two preference tests (AB test) by 7 listeners on speech quality
were conducted in the same manner as in the Section IV-B.
The evaluated synthetic speech samples are generated from therich context models with using 1) Clus, 2) Proposed (a = 0.1),

Copyright (c) 2014 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author’s version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/JSTSP.2013.2288599
JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012

-70

57

Log likelihood

56

Log likelihood

Target-80

55
54 Clus
Proposed

53
52

-90

-110

-130

50

-140
0.3 0.5 0.7
MDL factor a

0.9(a) HMM likelihood for natural parameters

0.3 0.5 0.7 0.9
MDL factor a(b) GV likelihood for generated parameters

Fig. 12. Objective evaluations for initialization method (spectrum).

3.05

3.8

Log likelihood

3.2
3

Clus

2.8

Log likelihood

3.00

3.4

Proposed

2.95

Proposed

Clus

2.90

2.6
2.4
0.9

Target2.85
0.1 0.3 0.5 0.7 0.9
MDL factor a(a) HMM likelihood for naturalparameters(b) GV likelihood for generated parameters

2.2
0.1

0.3 0.5 0.7
MDL factor a

2

40

F0

0
0.5

0.7

0.9

20
0.3

0.5
0.7
MDL factor a

0.9

5.2
5

Proposed

4.8
4.6
4.4

Clus

4.2
0.1

Target3.6

4

60

Fig. 14. Size of the decision trees for initial parameter generation.

Clus

-150
0.1

6

80 Spectrum

0
0.1

-120

51
49
0.1

Proposed-100

Tree size (%)

Target

58

100

-60

U/V error (%)

59

10

Fig. 13. Objective evaluations for initialization method (F0 ).

3) Proposed (a = 0.5), and 4) Target as the initial parameterfor the spectral parameter. As described above, a = 0.1 is thesetting of the highest GV likelihood and a = 0.5 is that of thehighest HMM likelihood. For the F0 component, they were
1) Conv: generated from conventional clustered model, andgenerated from the rich context models with using 2) Clus, 3)
Proposed (a = 0.6), and 4) Target as the initial parameters.
Note that the conventional clustered models were used for thecomponent that the proposed methods were not applied to.
The result of the preference test in the spectral component
is shown in Fig. 16, and that in the F0 component is shownin Fig. 17. From Fig. 16, the proposed initialization method
significantly improves speech quality compared with the conventional initialization method “Clus.” We can also see that thescore of “Proposed (a = 0.1)” is higher than that of “Proposed(a = 0.5).” This tendency is the same as that observed in theGV likelihood shown in Fig. 12(b). From Fig. 15 and Fig.

0.3

0.5
0.7
MDL factor a

0.9

Fig. 15. Error rates of the unvoiced/voiced decision.

17, although the setting of the parameter a to maximize theGV likelihood slightly increases U/V error rate, it was stillobserved to be effective to improve speech quality even in theF0 component.
F. Evaluation in Full Synthesis
To investigate the effectiveness of all proposed methods, weevaluated 5 kinds of synthetic speech1 shown in Table II. A
preference test (AB test) on speech quality was conducted by
8 listeners in the same manner as in the Section IV-B. Notethat “Target” was generated by parameter generation with richcontext models using the natural speech parameter sequence
as initial parameters.
The result of the preference test in full synthesis is shownin Fig. 19, and spectrograms of “Conventional,” “Proposed,”
and the natural speech are shown in Fig. 18. It is observed thata larger speech-quality improvement was yielded by applying
the proposed method to the spectral component than to the
TABLE IIS YNTHETIC SPEECH SAMPLES USED FOR “ FULL SYNTHESIS ”
EVALUATION .
Method
CC
CP
PC
PP
TT

Spectrum
Conventional
Conventional
Proposed (a = 0.1)
Proposed (a = 0.1)
TargetF0
Conventional
Proposed (a = 0.6)
Conventional
Proposed (a = 0.6)
Target1 Some samples are available from http://isw3.naist.jp/∼shinnosuke-t/
sample rcm.html

Copyright (c) 2014 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author’s version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/JSTSP.2013.2288599
JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 20121

11

Preference score

0.8

4

0.6

8

0.2
Clus

a=0.5
a=0.1
Proposed

TargetFrequency (kHz)0.4

0Conventional

095% confidence interval

Fig. 16. Preference scores on speech quality for investigating the effectivenessof initialization method for spectral component.

Proposed048

Natural0481Preference score95% confidence interval

0.8

Fig. 18. An example of spectrogram of synthetic speeches (“Natural”
represents the spectrograms of natural speech).

0.6
1
0.8

0.2
0

95% confidence interval

Preference score

0.4

0.6

Conv

Clus

a=0.6
Proposed

TargetFig. 17. Preference scores on speech quality for investigating the effectivenessof initialization method for F0 component.

F0 component. Moreover, a further improvement is yieldedby applying the proposed method to both spectral and F0
components, and the resulting speech quality shown as “PP”
is close to “TT.” From this result, we can see that the proposedparameter generation with rich context models for spectraland F0 components is very effective in improving quality insynthetic speech.
V. C ONCLUSION
In this paper, we proposed parameter generation methods
using rich context models in HMM-based speech synthesis
as yet another hybrid method combining HMM-based speechsynthesis and unit selection synthesis. In training, the rich context models were reformulated as Gaussian Mixture Models(GMMs). In synthesis, an initial speech parameters were generated from probability distributions over-fitted to individual
segments, and the speech parameter sequence was iteratively
generated from GMMs using a parameter generation method
based on the maximum likelihood criterion. The experimental
results have demonstrated: (1) the use of approximation witha single Gaussian component sequence yields better synthetic
speech quality than the use of EM algorithm in the proposedparameter generation method, (2) the state-based model selection yields quality improvements as same as the framebased model selection. (3) the proposed initialization method
is very effective to further improvement speech quality, and (4)the proposed methods for spectral and F0 components yields
significant improvements in synthetic speech quality comparedwith the traditional HMM-based speech synthesis.

0.4
0.2
0

CC

CP

PC

PP

TT

Fig. 19. Preference scores on speech quality for full synthesis.

ACKNOWLEDGMENT
Part of this work was supported by JSPS KAKENHI Grant
Number 24240032, and was executed under the Commissioned
Research for “Research and Development on Medical Communication Support System for Asian Languages based on
Knowledge and Language Grid” of National Institute of Information and Communications Technology (NICT), Japan. The
authors are grateful to Prof. Hideki Kawahara of Wakayama
University, Japan, for permission to use the STRAIGHT
analysis-synthesis method.
R EFERENCES[1] Y. Sagisaka, “Speech synthesis by rule using an optimal selection of
non-uniform synthesis units,” in Proc. ICASSP, New York, U.S.A, Apr.
1988, pp. 679–682.
[2] N. Iwahashi, N. Kaiki, and Y. Sagisaka, “Speech segment selection
for concatenative synthesis based on spectral distortion minimization,”
IEICE Trans., Fundamentals, vol. E76-A, no. 11, pp. 1942–1948, 1993.
[3] A. J. Hunt and A. Black, “Unit selection in a concatenative speechsynthesis system using a large speech database,” in Proc. ICASSP,
Atlanta, U.S.A., May 1996, pp. 373–376.
[4] A. K. Syrdal, C. W. Wightman, A. Conkie, Y. Stylianou, M. Beutnagel,
J. Schroeter, V. Strom, K.-S. Lee, and M. Makashay, “Corpus-based
techniques in the AT&T NextGen synthesis system,” in Proc. ICSLP,
Beijing, China, Oct 2000, pp. 410–415.
[5] H. Zen, K. Tokuda, and A. Black, “Statistical parametric speech synthesis,” Speech Commun., vol. 51, no. 11, pp. 1039–1064, 2009.
[6] T. Yoshimura, T. Masuko, K. Tokuda, T. Kobayashi, and T. Kitamura,“Speaker interpolation for HMM-based speech synthesis system,” J.
Acoust. Soc. Jpn. (E), vol. 21, no. 4, pp. 199–206, 2000.

Copyright (c) 2014 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author’s version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/JSTSP.2013.2288599
JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012[7] J. Yamagishi and T. Kobayashi., “Average-voice-based speech synthesis
using HSMM-based speaker adaptation and adaptive training,” IEICE
Trans., Inf. and Syst., vol. E90-D, no. 2, pp. 533–543, 2007.
[8] T. Nose, J. Yamagishi, T. Masuko, and T. Kobayashi, “A style control
technique for HMM-based expressive speech synthesis,” IEICE Trans.,
Inf. and Syst., vol. E90-D, no. 9, pp. 1406–1413, 2007.
[9] T. Toda and K. Tokuda, “A speech parameter generation algorithm
considering global variance for HMM-based speech synthesis,” IEICE
Trans., vol. E90-D, no. 5, pp. 816–824, 2007.
[10] S. King and V. Karaiskos, “The blizzard challenge 2011,” in Proc.
Blizzard Challenge workshop, Turin, Italy, Sept. 2011.
[11] Z. Ling, L. Qin, H. Lu, Y. Gao, L. Dai, R. Wang, Y. Jiang, Z. Zhao,
J. Yang, J. Chen, and G. Hu, “The USTC and iflytek speech synthesis
systems for blizzard challenge 2007,” in Proc. Blizzard Challenge
workshop, Bonn, Germany, Aug. 2007.
[12] Z. Yan, Q. Yao, and S. K. Frank, “Rich context modeling for highquality HMM-based TTS,” in Proc. INTERSPEECH, Brighton, U.K.,
Sept. 2009, pp. 1755–1758.
[13] Y. Qian, Z. Yan, Y. Wu, and F. K. Soong, “An HMM trajectory tiling(HTT) approach to high quality TTS,” in Proc. INTERSPEECH, Chiba,
Japan, Sept. 2010, pp. 422–425.
[14] T. Toda, H. Kawai, and M. Tsuzaki, “Optimizing sub-cost functions
for segment selection based on perceptual evaluations in concatenative
speech synthesis,” in Proc. ICASSP, Montreal, Canada, May 2004, pp.
657–660.
[15] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura,“Simultaneous modeling of spectrum, pitch and duration in HMM-based
speech synthesis,” in Proc. EUROSPEECH, Budapest, Hungary, Apr.
1999, pp. 2347–2350.
[16] K. Shinoda and T. Watanabe, “MDL-based context-dependent subword
modeling for speech recognition,” J. Acoust. Soc. Jpn.(E), vol. 21, no. 2,
pp. 79–86, 2000.
[17] K. Tokuda, T. Masuko, B. Miyazaki, and T. Kobayashi, “Multi-spaceprobability distribution HMM,” IEICE Trans., Inf. and Syst., vol. E85D, no. 3, pp. 455–464, 2002.
[18] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura,“Speech parameter generation algorithms for HMM-based speech synthesis,” in Proc. ICASSP, Istanbul, Turkey, June 2000, pp. 1315–1318.
[19] S. Kataoka, N. Mizutani, K. Tokuda, and T. Kitamura, “Decision treebacking-off in HMM-based speech synthesis,” in Proc. INTERSPEECH,
vol. 2, Jeju, Korea, Oct. 2004, pp. 1205–1208.
[20] Z. Ling and R. Wang, “HMM-based unit selection using frame sized
speech segments,” in Proc. INTERSPEECH, Pittsburgh U.S.A., Sept.
2013.
[21] T. Mizutani and T. Kagoshima, “Concatenative speech synthesis basedon the plural unit selection and fusion method,” IEICE Trans. on Inf.
and Syst., vol. E88-D, no. 11, pp. 2565–2572, 2005.
[22] C. J. Leggetter and P. C. Woodland, “Maximum likelihood linear
regression for speaker adaptation of continuous density hidden Markov
models,” in Computer Speech and Language, vol. 9, 1995, pp. 171–185.
[23] T. Shinozaki and M. Ostendorf, “Cross-validation and aggregated EM
training for robust parameter estimation,” in Computer Speech andLanguage, vol. 22, 2008, pp. 185–195.
[24] H. Zen, K. Tokuda, T. K. T. Masuko, and T. Kitamura, “Hidden semimarkov model based speech synthesis system,” IEICE Trans., Inf. andSyst., E90-D, no. 5, pp. 825–834, 2007.
[25] Y. Sagisaka, K. Takeda, M. Abe, S. Katagiri, T. Umeda, andH. Kuawhara, “A large-scale Japanese speech database,” in ICSLP90,
Kobe, Japan, Nov. 1990, pp. 1089–1092.
[26] H. Kawahara, J. Estill, and O. Fujimura, “Aperiodicity extraction andcontrol using mixed mode excitation and group delay manipulation
for a high quality speech analysis, modification and synthesis system
STRAIGHT ,” in MAVEBA 2001, Firentze, Italy, Sept. 2001, pp. 1–6.
[27] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Maximum likelihood voice conversion based on GMM with STRAIGHT mixed excitation,” in Proc. INTERSPEECH, Pittsburgh, U.S.A., Sep. 2006, pp.
2266–2269.
[28] H. Kawahara, I. Masuda-Katsuse, and A. D. Cheveigne, “Restructuring
speech representations using a pitch-adaptive time-frequency smoothing
and an instantaneous-frequency-based F0 extraction: Possible role of arepetitive structure in sounds,” Speech Commun., vol. 27, no. 3–4, pp.
187–207, 1999.

12

Shinnosuke Takamichi received his B.E. from Nagaoka University of Technology, Japan, in 2011and his M.E. degree from the Graduate Schoolof Information Science, Nara Institute of Scienceand Technology (NAIST), Japan, in 2013. He iscurrently a Ph. D. student of NAIST. His researchinterests include electroacoustics, signal processing,and speech synthesis. He is a student member of
ASJ, and is a member of ISCA.

Tomoki Toda earned his B.E. degree from Nagoya
University, Aichi, Japan, in 1999 and his M.E. and
D.E. degrees from the Graduate School of Information Science, NAIST, Nara, Japan, in 2001 and 2003,
respectively. He was a Research Fellow of JSPS in
the Graduate School of Engineering, Nitech, Aichi,
Japan, from 2003 to 2005. He was an AssistantProfessor of the Graduate School of Information
Science, NAIST from 2005 to 2011, where he iscurrently an Associate Professor. He has also been a
Visting Researcher at the NICT, Kyoto, Japan, since
May 2006. From March 2001 to March 2003, he was an Intern Researcher at
the ATR SLC Research Laboratories, Kyoto, Japan, and then he was a Visiting
Researcher until March 2006. He was also a Visiting Researcher at the LTI,
CMU, Pittsburgh, USA, from October 2003 to September 2004 and at the
CUED, Cambridge, UK, from March to August 2008. His research interests
include statistical approaches to speech processing. He received the 18th
TELECOM System Technology Award for Students and the 23rd TELECOM
System Technology Award from the TAF, the 2007 ISS Best Paper Award andthe 2010 ISS Young Researcher’s Award in Speech Field from the IEICE,
the 10th Ericsson Young Scientist Award from Nippon Ericsson K.K., the
4th Itakura Prize Innovative Young Researcher Award and the 26th Awaya
Prize Young Researcher Award from the ASJ, the 2009 Young Author Best
Paper Award from the IEEE SPS, the Best Paper Award (Short Paper inRegular Session Category) from APSIPA ASC 2012, the 2012 Kiyasu Special
Industrial Achievement Award from the IPSJ, and the 2013 Best Paper Award(Speech Communication Journal) from EURASIP-ISCA. He was a memberof the Speech and Language Technical Committee of the IEEE SPS from2007 to 2009.

Yoshinori Shiga has been involved in speech technology research, since 1987, at various institutions
including the University of Edinburgh, Edinburgh,
U.K., the University of Surrey, Guildford, U.K., theTokyo University of Science, Tokyo, Japan, Toshiba
Corporation, Kawasaki, Japan, Advanced Telecommunications Research Institute International (ATR),
Kyoto, Japan, and the National Institute of Information and Communications Technology (NICT),
Kyoto, Japan. Currently he is the leader of the Spoken Dialogue & Speech Synthesis Team at NICT’s
Spoken Language Communication Group. He holds B.E. and M.E. degrees
in Electrical Engineering from the Tokyo University of Science, and a Ph.D.
degree in Speech Technology from the University of Edinburgh. His researchinterests are in the area of speech and sound processing including speechsynthesis and recognition. From 2009 to 2011, he was a Guest Professor of
the Graduate School of Intercultural Studies at Kobe University, Kobe, Japan.
He received the 2002 Information and Systems Society Paper Award fromthe Institute of Electronics, Information and Communication Engineers. He
is currently a member of International Speech Communication Association(ISCA), the Acoustical Society of Japan (ASJ) and the Phonetic Society ofJapan.

Copyright (c) 2014 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author’s version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/JSTSP.2013.2288599
JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 201213

Sakriani Sakti received her B.E degree in Informatics (cum laude) from Bandung Institute ofTechnology, Indonesia, in 1999. In 2000, she received ”DAAD-Siemens Program Asia 21st Century” Award to study in Communication Technology,
University of Ulm, Germany, and received her MSc
degree in 2002. During her thesis work, she worked
with Speech Understanding Department, DaimlerChrysler Research Center, Ulm, Germany. Between
2003-2009, she worked as a researcher at ATR SLC
Labs, Japan, and during 2006-2011, she worked as
an expert researcher at NICT SLC Groups, Japan. While working with ATRNICT, Japan, she continued her study (2005-2008) with Dialog Systems Group
University of Ulm, Germany, and received her PhD degree in 2008. She
actively involved in collaboration activities such as Asian Pacific Telecommunity Project (2003-2007), A-STAR and U-STAR (2006-2011). She also
served as a visiting professor of Computer Science Department, Universityof Indonesia (UI) in 2009-2011. Currently, she is an assistant professor of
the Augmented Human Communication Lab, NAIST, Japan. She is a memberof JNS, SFN, ASJ, ISCA, IEICE amd IEEE. Her research interests include
statistical pattern recognition, speech recognition, spoken language translation,
cognitive communication, and graphical modeling framework.

Graham Neubig received his B.E. from University
of Illinois, Urbana-Champaign in 2005, and his M.S.
and Ph.D. in informatics from Kyoto University in2010 and 2012 respectively. From 2012, he has been
an assistant professor at the Nara Institute of Scienceand Technology, where he is pursuing research inmachine translation and spoken language processing.

Satoshi Nakamura is Professor of Graduate Schoolof Information Science, Nara Institute of Science andTechnology, Japan, Honorarprofessor of Karlsruhe
Institute of Technology, Germany, and ATR Fellow.
He received his B.S. from Kyoto Institute of Technology in 1981 and Ph.D. from Kyoto University
in 1992. He was Associate Professor of Graduate
School of Information Science at Nara Instituteof Science and Technology in 1994-2000. He was
Director of ATR Spoken Language Communication
Research Laboratories in 2000-2008 and Vice president of ATR in 2007-2008. He was Director General of Keihanna ResearchLaboratories and the Executive Director of Knowledge Creating Communication Research Center, National Institute of Information and Communications
Technology, Japan in 2009-2010. He is currently Director of Augmented
Human Communication laboratory and a full professor of Graduate Schoolof Information Science at Nara Institute of Science and Technology. He also
serves as a visiting professor of Collaborative Research Unit, National Instituteof Informatics. He is interested in modeling and systems of speech-to-speechtranslation and speech recognition. He is one of the leaders of speech-tospeech translation research and has been serving for various speech-to-speechtranslation research projects in the world including C-STAR, IWSLT and
A-STAR. He was a project leader of the world first network-based commercial speech-to-speech translation service for 3-G mobile phones in 2007
and VoiceTra project for iPhone in 2010. He received Yamashita ResearchAward, Kiyasu Award from the Information Processing Society of Japan,
Telecom System Award, AAMT Nagao Award, Docomo Mobile ScienceAward in 2007, ASJ Award for Distinguished Achievements in Acoustics.
He received the Commendation for Science and Technology by the Ministerof Education, Science and Technology, and the Commendation for Scienceand Technology by the Minister of Internal Affair and Communications.
He also received LREC Antonio Zampoli Award 2012. He organized theInternational Workshop of Spoken Language Translation (IWSLT 2006) and
Oriental Cocosda 2008 as a general chair. He also served as the program chairof INTERSPEECH 2010. He has been Elected Board Member of International
Speech Communication Association, ISCA, since June 2011 and IEEE Signal
Processing Magazine Editorial Board Member since April 2012.

Copyright (c) 2014 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.