8th ISCA Speech Synthesis Workshop • August 31 – September 2, 2013 • Barcelona, Spain

Text-to-speech synthesizer based on combination of composite wavelet andhidden Markov models
Nobukatsu Hojo1 , Kota Yoshizato1 , Hirokazu Kameoka1,2 ,Daisuke Saito1 , Shigeki Sagayama1,∗
1

Graduate School of Information Science and Technology, the University of Tokyo, Japan
2
Communication Science Laboratories, NTT Corporation, Japan{hojo, yoshizato, kameoka, dsaito, sagayama}@hil.t.u-tokyo.ac.jp

Abstract
This paper proposes a text-to-speech synthesis (TTS) system based on a combined model consisting of the Composite Wavelet Model (CWM) and the Hidden Markov Model(HMM). Conventional HMM-based TTS systems using cepstral features tend to produce over-smoothed spectra, which
often result in mufﬂed and buzzy synthesized speech. Thisis simply caused by the averaging of spectra associated witheach phoneme during the learning process. To avoid the oversmoothing of generated spectra, we consider it important to focus on a different representation of the generative process ofspeech spectra. In particular, we choose to characterize speech
spectra using the CWM, whose parameters correspond to thefrequency, gain and peakiness of each underlying formant. Thisidea is motivated by our expectation that the averaging of theseparameters would not lead directly to the over-smoothing ofspectra, as opposed to the cepstral representations. To describe
the entire generative process of a sequence of speech spectra, we
combine the generative process of a formant trajectory using an
HMM and the generative process of a speech spectrum using
the CWM. A parameter learning algorithm for this combinedmodel is derived based on an auxiliary function approach. We
conﬁrmed through experiments that our speech synthesis system was able to generate speech spectra with clear peaks anddips, which resulted in natural-sounding synthetic speech.
Index Terms: text-to-speech synthesis, hidden Markov model,
composite wavelet model, formant, Gaussian mixture model,
auxiliary function

1. Introduction
This paper proposes a new model for text-to-speech synthesis (TTS). One promising approach for TTS involves methods
based on statistical models. In this approach, the ﬁrst step isto formulate a generative model of a sequence of speech features. The second step is to train the parameters of the assumed
generative model given a set of training data in a speech corpus. The third step is to produce the most likely sequence ofspeech features given a text input and transform it into a speech
signal. With this approach, one key to success is that the assumed generative model reﬂects the nature of real speech well.
To model the entire temporal evolution of speech features, ahidden Markov model (HMM) and its variants including the“trajectory HMM” have been introduced with notable success[1, 2, 3]. HMMs are roughly characterized by the structure of
the state transition network (i.e., a transition matrix) and an output distribution assumption. In conventional HMM-based TTS
systems, a Gaussian mixture emission distribution of a cepstralfeature [1, 2] or a line spectrum pair (LSP) feature [4, 3] is typically used as the output distribution of each state. The aim of
this paper is to seek an alternative to the conventional speechfeature and the state output distribution. Namely, this paper isconcerned with formulating a new model for the generative process of speech spectra and combining it with an HMM.
The Gaussian distribution assumption of a cepstral featuredescribes probabilistic ﬂuctuations of spectra in the power direction, while that of an LSP feature describes probabilistic
ﬂuctuations of spectral peaks in the frequency direction. Since
the frequencies and powers of peaks in a spectral envelope correspond to the resonance frequencies and gains of the vocal
tract, they both vary continuously over time according to thephysical movement of the vocal tract. In particular, resonancefrequencies and gains both vary signiﬁcantly in the boundary
between one phoneme and another. To achieve higher quality
speech synthesis, we consider it important to describe a generative model that takes account of the ﬂuctuations of spectral peaks in both the frequency and power directions rather
than a ﬂuctuation in just one direction. Conventional HMMbased TTS systems using cepstral features tend to produce oversmoothed spectra, which often result in mufﬂed and buzzy synthesized speech. This is simply caused by the averaging of observed log-spectra assigned to each state during the training process (Fig. 2 (a)). Although some attempts were made to emphasize the peaks and dips of generated spectra via post-processing[5], it is generally difﬁcult to restore original peaks and dips
once spectra are over-smoothed. By contrast, this paper proposes tackling this problem by introducing a different spectral
representation called the Composite Wavelet Model (CWM)[6, 7, 8] as an alternative to cepstral and LSP representations,on which basis we formulate a new generative model for TTS.

2. Generative model of spectral sequence
2.1. Motivation
Taking the mean of cepstral features associated with a particular phoneme amounts to taking the mean of the corresponding
log spectra. Since the resonance frequencies and gains of the
vocal tract both vary continuously during the transition fromone phoneme to another, if we simply take the mean of the log
spectra, the spectral peaks and dips will be be blurred and indistinct (Fig. 2 (a)). By contrast, if we can appropriately treat thefrequencies and powers of individual spectral peaks as speechfeatures, we expect that taking their means will not cause theblurring of spectra (Fig. 2 (b)). CWM approximates a speech
spectral envelope from the sum of the Gaussian distributions,129

Figure 1: Spectral representation based on CWM

interpreted as a function of frequency (see Fig. 1 for a graphicalillustration). This means each Gaussian distribution function
roughly corresponds to a peak in a spectral envelope. CWM
is thus convenient for describing both the frequency and power
ﬂuctuations of spectral peaks.
Another notable feature of CWM is that its parameters canbe easily transformed into a signal. Since a Gaussian functionin the frequency domain corresponds to a Gabor function in thetime domain, CWM parameters can be directly converted into
a signal by superimposing the corresponding Gabor functions.
CWM can thus be regarded as a speech synthesizer with an FIR
ﬁlter and its superiority to conventional systems with IIR ﬁlters
has already been shown in [6].
One straightforward way of incorporating CWM into an
HMM-based TTS system would be to ﬁrst perform CWM parameter extraction on a frame-by-frame basis, and then train
the parameters of an HMM by treating the extracted CWM parameters as a sequence of feature vectors. However, our preliminary investigation revealed that this simple method did notwork well [9]. One reason for this is the common difﬁcultyof formant tracking. Although formants and their time courses(formant trajectories) are clearly visible to the human eye in the
spectrum and in the spectrogram, automatic formant extraction
and tracking are far from trivial. Many formant extraction algorithms miss a formant that is present, insert a formant when
there is none, or mislabel them (such as label F1 as F2 , or F3
as F2 ), mostly as a result of incorrectly pruning the correct formant at the frame level. This is also the case for frame-by-frame
CWM parameter extraction since it can be thought of as a sortof formant extraction. Fig. 3 shows an example of results obtained with frame-by-frame CWM analysis. As this example
shows, the index of each Gaussian distribution function in the
CWM at one particular frame is not always consistent with thatat another frame. This was problematic when training the HMM
parameters since the mean of the emission distribution of eachstate is given as the average of the feature vectors assigned tothe same state. To train the HMM parameters appropriately, theindices of the Gaussian functions of the CWM assigned to thesame state must always be consistent. This implies the need forthe joint modeling of the CWM and HMM. Motivated by theabove discussion, we here describe the entire generative process of a sequence of speech spectra by combining the generative process of a speech spectrum based on the CWM with thegenerative process of a CWM parameter trajectory based on an
HMM.
2.2. Formulation
The CWM consists of parameters (roughly) corresponding to
the frequency and power of spectral peaks. Speciﬁcally, theCWM approximates a spectral envelope by employing a Gaussian mixture model (GMM) interpreted as a function of fre-

130

power

power

N. Hojo, K. Yoshizato, H. Kameoka, D. Saito, S. Sagayama

frequency

frequency(a) Mean in cepstral domain(b) Mean in CWM domain

Figure 2: Expectations of spectra taken in different domains.
The dashed lines indicate the training samples and the solidlines indicate the mean spectra.
quency. Namely, the CWM fω,l is deﬁned asfω,l =

K
X
k=1

√

wk,l
exp
2πσk,l„−«(ω − µk,l )2
,
2
2σk,l

(1)

where ω, l denotes the indices of frequency and time, respectively, and K is the number of mixture components of the
GMM. See Fig. 1 for a graphical illustration. µk,l , σk,l and wk,lare the mean, variance and weight of each Gaussian (when interpreted as a probability density function), and thus correspondto the frequency, peakiness and power of the peaks in a spectralenvelope, respectively.
Using the CWM representation given above as a basis, here
we describe the generative process of an entire sequence of observed spectra. We consider an HMM that generates a set con2
sisting of µk,l , ρk,l := 1/σk,l, and wk,l at time l (see Fig. 4).
Each state of the HMM represents a label indicating linguistic information, which can be simply either a phoneme label asshown in Fig. 4 or a context label (as used in [10]). Given state
sl at time l, we assume that the CWM parameters are generated
according to2),
P (µk,l |sl ) = N (µk,l ; mk,sl , ηk,s
l

(ρ)
(ρ)
P (ρk,l |sl ) = Gamma(ρk,l ; ak,sl , bk,sl ),(w)(w)
P (wk,l |sl ) = Gamma(wk,l ; ak,sl , bk,sl ),

(2)
(3)
(4)

where N (x; m, η 2 ) denotes the normal distribution andGamma(x; a, b) the gamma distribution:
Gamma(x; a, b) = xa−1

exp(−x/b).
Γ(a) ba(5)

These distribution families are chosen for simplifying thederivation of the parameter estimation algorithm described in
the next section. Given the following sequence of CWM parameters, — = {µk,l }k,l , ȷ = {ρk,l }k,l , w = {wk,l }k,l , wefurther assume that a spectrum {yω,l }ω observed at time l is
generated according to
P (yω,l |—, ȷ, w) = Poisson(yω,l ; fω,l ),(6)where Poisson(x; λ) denotes the Poisson distribution:
Poisson(x; λ) =λx e−λ.
x!

(7)This assumption is also made for simplifying the derivation of
the parameter estimation algorithm described in the next section. fω,l denotes the sequence of spectrum models deﬁned by8th ISCA Speech Synthesis Workshop • August 31 – September 2, 2013 • Barcelona, SpainĨƌĞƋƵĞŶĐǇ

4

2

0
0

0.1

0.2
0.3
Time (s)0.4

0.5(a) The frame-independent CWM parameter extraction
method [9]⊹∱∻≬ ≷∱∻≬≷⊹∲∻≬ ∲∻≬

Frequency (kHz)∲ ⊹ ⊻ ≎∨≭ ∻⊺∲ ∩ ∳≫∻≬≫∻≳≬ ≫∻≳≬∨⊽∩ ∷≆∡∻≬ ∶∶∴ ⊽≫∻≬ ⊻ ≇≡≭≭≡∨≡∨⊽∩≫∻≳≬ ∻≢≫∻≳≬ ∩∷∵∨≷∩≷≫∻≬ ⊻ ≇≡≭≭≡∨≡∨≷∩≫∻≳≬ ∻≢≫∻≳≬ ∩

ƚŝŵĞ

≙∡∻≬

≙∡∻≬ ⊻ ≐≯≩≳≳≯≮∨≆∡∻≬∩

ƚŝŵĞ
Figure 4: Illustration of the present HMM

8

6

4

2

0
0

≳≬

ͬŝͬ

ͬĂͬ

6

ĨƌĞƋƵĞŶĐǇ

Frequency (kHz)

8

0.1

0.2
0.3
Time (s)0.40.5(b) The proposed method

Figure 3: An example of the formant frequencies extractedusing (a) the frame-independent CWM parameter extraction
method [9] and (b) the proposed method. The extracted formant frequencies are plotted with different colors according to
the indices of the Gaussian distribution functions in CWM.
Eq. 1. It should be noted that the maximization of the Poisson
likelihood with respect to fω,l amounts to optimally ﬁtting fω,lto yω,l by using the I-divergence as the ﬁtting criterion [8]. The
next section describes the parameter estimation algorithm of thegenerative model presented above.

where α is a regularization parameter that weighs the imporc
tance of the log prior density relative to the log-likelihood. =denotes equality up to constant terms. Unfortunately, this optimization problem is non-convex, and ﬁnding the global optimum is an intractable problem. However, we can employ an
auxiliary function approach to ﬁnd a local optimum [8].
As mentioned above, we notice from Eqs. (6) and (7) that− log P (Y |Θ) is equal up to constant terms to the sum of theI-divergence between yω,l and fω,l
”
X“
yω,l
− yω,l + fω,l .
(10)I(Θ) :=
yω,l logfω,lω,lHence, maximizing P (Θ|Y ) amounts to minimizing I(Θ) −α log P (Θ) with respect to Θ. By invoking Jensen’s inequality
based on the concavity of the logarithm function, we obtain an
inequality
”
X“
yω,l
− yω,l + fω,l(11)
I(Θ) =
yω,l log
fω,l
ω,l
Xn
≤
yω,l log yω,l − yω,lω,l−3. Parameter estimation algorithm
Here we describe the parameter learning algorithm given a
set of training data. The spectral sequence of the training data is considered to be an observed sequence and aset of unknown parameters of the present generative model
is denoted by Θ. Θ consists of the state sequence s ={sl }l , the parameters of the state emission distributions „ =(w) (w)(ρ) (ρ){mk,i , ηk,i , ak,i , bk,i , ak,i , bk,i }k,i , and the CWM parameters {—, ȷ, w}.
Given a set of observed spectra Y = {yω,l }ω,l , we would
like to determine the estimate of Θ that maximizes the posterior density P (Θ|Y ) ∝ P (Y |Θ)P (Θ), or equivalently,
log P (Y |Θ) + log P (Θ). Here, log P (Θ|Y ) is written asclog P (Θ|Y ) = log P (Y |Θ) + α log P (Θ),(8)

clog P (Θ) = log P (s) + log P (—|s, „)+ log P (ȷ|s, „) + log P (w|s, „),

(9)

”o
X“
gk,ω,lλk,ω,l log+ gk,ω,l ,
λk,ω,l

(12)

k

where
gk,ω,l =

r

ρk,l
exp
2π

„

«
ρk,l
2
−
(ω − µk,l ) .
2(13)

By using J (Θ, λ) to denote the upper bound of I(Θ), i.e., theright-hand side of (12), equality holds if and only if
wk,l gk,ω,l.
λk,ω,l = PK
j=1 wj,l gj,ω,l(14)

Now, when λk,ω,l is given by Eq. (14) with an arbitrary Θ, theauxiliary function J (Θ, λ) − α log P (Θ) becomes equal to theobjective function I(Θ) − α log P (Θ). Then, the parameterthat decreases J (Θ, λ)−α log P (Θ) while keeping λ ﬁxed will
necessarily decrease I(Θ) − α log P (Θ), since inequation (12)guarantees that the original objective function is always even131

N. Hojo, K. Yoshizato, H. Kameoka, D. Saito, S. Sagayama
smaller than the decreased J (Θ, λ) − α log P (Θ). Therefore,
by repeating the update of λ by Eq. (14) and the update of Θthat decreases J (Θ, λ) − α log P (Θ), the objective function
decreases monotonically and converges to a stationary point.
With ﬁxed s and „, the auxiliary function J (Θ, λ) −α log P (Θ) is minimized with respect to the CWM parameters,—, ȷ, and w, under the following updatesµk,l =

2
Dk,l ηk,iρk,l + αmk,i,2
Ck,l ηk,iρk,l + α

(15)

(ρ)

ρk,l =

Ck,l + 2α(ak,l − 1)(ρ)

Ck,l µ2k,l − 2Dk,l µk,l + Ek,l − 2α/bk,l

(16)

,

(w)

wk,l =

Ck,l + α(ak,l − 1)(w)

1 + α/bk,l

(17)

,

where
Ck,l =Xyω,l λk,ω,l ,(18)

ωyω,l λk,ω,l ,(19)

ω 2 yω,l λk,ω,l .

(20)

Figure 5: Spectrogram of a natural voice and the trajectories ofextracted frequency parameters (a Japanese sentence of A01 in
the ATR-503 data set).

ω

Dk,l =Xω

Ek,l =

X
ωWith ﬁxed —, ȷ, w and „, the state sequence minimizing
J (Θ, λ) − α log P (Θ) can be obtained by using the Viterbi
algorithm. With ﬁxed —, ȷ, w and s, the auxiliary function isminimized under the following updates.
2
As for {mk,i , νk,i
}k,i ,
mk,i =

1 X
µk,l
Ni

(21)

l∈Ti

2
νk,i
=

1 X 2
1 X
µk,l − (µk,l )2 ,
Ni
Ni
l∈Ti

(22)

l∈Ti

(ρ)

(ρ)

(w)

(w)

where Ti = {l|sl = i}. As for {ak,i , bk,i , ak,i }k,l , bk,i }k,l ,although the updates are not derived in a closed form, they are
updated as the root of the following equations
1
(ρ)
(ρ)
log ak,i − ψ(ak,i ) = log(
Ni

X
l∈Ti

1 X
ρk,l ) −
ρk,l
Ni
l∈Ti

(23)
X
X
1
1
(w)
(w)
log ak,i − ψ(ak,i ) = log(
wk,l ) −
wk,l
Ni
Ni
l∈Ti

(ρ)

bk,i =

1
Ni

1
N
(w)
bk,i = i

P

l∈Ti

(24)

l∈Ti ρk,l
(ρ)
ak,i

(25)

l∈Ti wk,l
,
(w)
ak,i

(26)

P

where ψ(a) denotes the digamma function
ψ(a) =∂Γ(a)/Γ(a).
∂a(27)

A spectrogram of sentence A01 from the ATR503 data set‘Arayurugenjitsuwo subete jibunnohohe nejimagetanoda’ andan example of the trajectory of the mean parameters of the
CWM extracted from it are shown in Fig. 5.

132

4. Speech Synthesis Experiment and
Evaluation
4.1. Evaluation criterion
To evaluate the spectral distortion caused by the training process, we measured the spectral distortion between the synthesized speech and real speech.
Bark spectral distortion [11] is known to be an objective
measure for incorporating psychoacoustic responses. We used
Bark spectral distortion and log spectral distortion to measure the spectral divergence between the synthesized and realspeech. Using this criterion, we compared the present methodwith HTS-2.1 [10], a conventional HMM speech synthesis system with mel-cepstrum features and the global variance model[?].
The time alignment between the two spectral sequences wascomputed by using the dynamic time warping (DTW) algorithm. The average distortions on 53 synthesized speech sentences were compared and a statistical test was conducted.
4.2. Experimental Conditions
All the phoneme boundaries of the training data were given by
the phoneme labels of HTS [10]. This means that the state sequences of the HMM are assumed to be known in the training stage. The number of the Gaussian functions in the CWM
was set at 24. The initial CWM parameters were determinedby performing the method reported by [9] to estimate the CWM
parameters from the entire spectral sequence corresponding toeach phoneme. We used an acoustic model of 1 state and a leftto-right HMM with no skip, as with HTS-2.1 [10]. We used
“STRAIGHT” [12] to compute the spectral envelopes of the
training data, with an analysis interval of 5 ms. We used 450
uttered sentences for the training and 53 uttered sentences forthe synthesis and evaluation, respectively. All the speech samples were uttered by a Japanese male speaker and recorded witha 16kHz sampling rate and a 16 bit quantization, which weretaken from the demo site of HTS [13].

8th ISCA Speech Synthesis Workshop • August 31 – September 2, 2013 • Barcelona, Spain

5. Conclusion
Normalized Distortion

1.05

CWM
mcep−gv
mcep−nongv

1
0.95
0.9
0.85
0.8

log spectral distortion

cepstral distortion

Figure 6: Normalized squared distances between the log spectra and cepstra of real and synthesized speech by the proposed
method (red) , the ordinary mel-cepstral method with gv (blue)
and without gv (green).
2
CWM
mcep
samplelog power0

6. References−2[1] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, andT. Kitamura, “Simultaneous modeling of spectrum, pitchand duration in HMM-based speech synthesis,” in Proc.
of Eurospeech 1999, 1999, pp. 2347–2350.

−4
−6
−8
−10
0

This paper proposed a text-to-speech synthesis (TTS) system based on a combined model consisting of the Composite Wavelet Model (CWM) and the Hidden Markov Model(HMM). To avoid the over-smoothing of spectra generated by
the conventional HMM-based TTS systems using cepstral features, we considered it important to focus on a different representation of the generative process of speech spectra. In particular, we chose to characterize the speech spectra with the CWM,
whose parameters correspond to the frequency, gain and peakiness of each underlying formant. This idea was motivated byour expectation that averaging these parameters would not directly cause the over-smoothing of the spectra, unlike the cepstral representations. To describe the entire generative processof a sequence of speech spectra, we combined the generative
process of a formant trajectory using an HMM and the generative process of a speech spectrum using the CWM. We developed a parameter learning algorithm for this combined model
based on an auxiliary function approach. We conﬁrmed throughexperiments that our speech synthesis system was able to generate speech spectra with clear peaks and dips, which resulted
in natural-sounding synthetic speech.

24Frequency(kHz)6

8

Figure 7: An example of the synthesized spectral envelope corresponding to the phoneme ’/e/’
4.3. Experimental Results
The measured spectral distortions are shown in Fig. 6. The distances were normalized according to the values obtained withthe proposed and conventional methods. We conﬁrmed througha statistical test that the Bark spectral distortion obtained withthe proposed method was signiﬁcantly smaller than that obtained with the conventional method, while the two methods
were comparable in terms of the log spectrum distortion. This
result can be interpreted as showing that the proposed model
was able to synthesize spectra properly especially in a low frequency region. Since the changes in the peak frequencies ofspectra are mainly seen in the low frequency region, this result indicates that the proposed model is superior to the conventional model as regards modeling ﬂuctuations in the frequency
direction. Fig. 7 shows an example of the synthesized spectralenvelope along with the spectral envelope of real speech corresponding to the phoneme ’/e/’. In addition to the superiorityof the proposed model in terms of the Bark spectral distortion,
the proposed method was able to restore spectral peaks and dips
more clearly than the conventional model, especially in the 1.5
to 6 kHz frequency range. In future we plan to conduct a subjective evaluation test to conﬁrm whether the perceptual qualityof the synthesized speech has actually been improved.

[2] H. Zen, K. Tokuda, and T. Kitamura, “An introduction of
trajectory model into HMM-based speech synthesis,” inProc. of 5th ISCA Speech Synthesis Workshop, 2004, pp.
191–196.
[3] Y. Qian, Z.-J. Yan, Y.-J. Wu, F. K. Soong, G. Zhang, andL. Wang, “An HMM trajectory tiling (HTT) approach tohigh quality TTS,” in Proc. of Interspeech 2010, 2010, pp.
422–425.
[4] F. Itakura, “Line spectrum representation of linear predictor coefﬁcients of speech signals,” J. Acoust. Soc. Am.,
vol. 57, no. 1, p. S35, 1975.
[5] T. Toda and T. Tokuda, “A speech parameter generation
algorithm considering global variance for HMM-based
speech synthesis,” IEICE Trans. Inf. & Syst., vol. E90-D,
no. 5, pp. 816–824, 2007.
[6] T. Saikachi, K. Matsumoto, S. Sako, and S. Sagayama,“Discussion about speech analysis and synthesis by composite wavelet model,” in Proc. ASJ Spring Meeting,vol. 2, 2006, pp. 89–94, in Japanese.
[7] P. Zolfaghari and T. Robinson, “Formant analysis using
mixtures of Gaussians,” in Proc. of ICSLP ’96, vol. 2,
1996, pp. 1229–1232.
[8] H. Kameoka, N. Ono, and S. Sagayama, “Speech spectrum modeling for joint estimation of spectral envelope
and fundamental frequency,” IEEE Trans. Audio, Speech& Language Process., vol. 18, no. 6, pp. 1507–1516,
2010.
[9] N. Hojo, K. Minami, D. Saito, H. Kameoka, and
S. Sagayama, “HMM speech synthesis using speech analysis based on composite wavelet model.” in Proc. ASJ Autumn Meeting, no. 2-7-7, 2013, in Japanese.

133

N. Hojo, K. Yoshizato, H. Kameoka, D. Saito, S. Sagayama[10] H. Zen, T. Nose, J. Yamagishi, S. Sako, T. Masuko, A. W.
Black, and K. Tokuda, “The HMM-based speech synthesis system version 2.0,” in Proc. Proc. of 6th ISCA Speech
Synthesis Workshop, vol. 6, 2007, pp. 294–299.
[11] S. Wan, A. Sekey, and A. Gersho, “An objective measurefor predicting subjective quality of speech coders,” IEEE
Journal on Selected Areas in Communications, vol. 10,
no. 5, pp. 819–829, 1992.
[12] H. Kawahara, I. Masuda-Katsuse, and A. Cheveigne, “Restructuring speech representations using a pitch-adaptive
time-freqency smoothing and an instantaneous-frequencybased f0 extraction,” Speech Communication, vol. 27, no.
3–4, pp. 187–207, 1999.
[13] http://hts.sp.nitech.ac.jp/.

134