INTERSPEECH 2014

Speech Prosody Generation for Text-to-Speech Synthesis Based on
Generative Model of F0 Contours
Kento Kadowaki1 , Tatsuma Ishihara1 , Nobukatsu Hojo1 , Hirokazu Kameoka1,2
1

Graduate School of Information Science and Technology, The University of Tokyo, Japan
2
NTT Communication Science Laboratories, NTT Corporation, Japan{kadowaki,ishihara,hojo,kameoka}@hil.t.u-tokyo.ac.jp

Abstract

assigned context labels. Thus, the Fujisaki model can potentially be a good model for F0 contour synthesis. However,since the Fujisaki model does not take the form of a statistical (automatically trainable) model, using the Fujisaki model
for synthesizing F0 contours within a statistical framework is
not straightforward. Indeed, estimating (learning) the Fujisaki
model parameters from raw F0 contour observations has been adifficult task. Several techniques have already been developed(e.g., [3, 4, 5, 6]), but so far with limited success due to the difficulty in searching for optimal parameters under the constraints
imposed in the Fujisaki model.
We have previously formulated a statistical model of speech
F0 contours by translating the Fujisaki model into a probabilistic model described as a discrete-time stochastic process[7, 8]. This formulation has allowed us not only to derive an efficient parameter inference algorithm utilizing powerful statistical methods but also to obtain an automatically trainable versionof the Fujisaki model. The aim of this paper is to further extend
this model to a context-dependent one so as to be able to learn
and generate the Fujisaki parameters from input sentences.
The rest of this paper is organized as follows. Sec. 2 briefly
reviews the original Fujisaki model and a discrete-time stochastic counterpart to the Fujisaki model, that we have introduced in[7, 8]. Sec. 3 proposes to extend it to a context-dependent one.
Sec. 4 proposes a parameter training algorithm for the present
model based on decision tree-based context clustering. Sec. 5shows some results of a speech synthesis experiment conductedusing real speech data excerpted from the ATR speech database.
Sec. 6 concludes this paper.

This paper deals with the problem of generating the fundamental frequency (F0 ) contour of speech from a text input
for text-to-speech synthesis. We have previously introduced
a statistical model describing the generating process of speech
F0 contours, based on the discrete-time version of the Fujisaki
model. One remarkable feature of this model is that it has allowed us to derive an efficient algorithm based on powerful statistical methods for estimating the Fujisaki-model parametersfrom raw F0 contours. To associate a sequence of the Fujisakimodel parameters with a text input based on statistical learning,
this paper proposes extending this model to a context-dependentone. We further propose a parameter training algorithm for thepresent model based on a decision tree-based context clustering.
Index Terms: Speech F0 contours, stochastic model, Fujisaki
model, hidden Markov model, EM algorithm

1. Introduction
The fundamental frequency (F0 ) contour of speech is a feature that represents the intonation of an utterance. One important challenge of the text-to-speech synthesis research involves
developing a reasonable model and method for generating anatural-sounding F0 contour from a text input.
Thanks to the increasing availability of speech databases,
speech synthesis systems based on statistical models such as
hidden Markov models (HMMs) have attracted particular attention in recent years. In the HMM-based speech synthesis system[2], a sequence of spectra, F0 s and their delta and acceleration
components is modeled simultaneously within a unified HMM
framework. At the synthesis stage, a sequence of these parameters is generated according to the output probabilities of the
trained HMM given an input sentence. The constraints of the
dynamic parameters are considered during parameter generation in order to guarantee the smoothness of the generated spectral and F0 trajectories. However, conventional HMM based
speech synthesis systems tend to produce over-smoothed F0
contours, which often result in a synthesis that sounds “emotionless” to human listeners.
In speech synthesis technology, one important challenge
involves synthesizing an F0 contour that is not only linguistically appropriate but also physically likely to be generated
via the control mechanism of phonation. The Fujisaki model[1] is a well-founded mathematical model, that describes theprocess by which the whole F0 contour of a speech utterance
is generated. This model is known to approximate actual F0
contours of speech well when the parameters are chosen appropriately. In the Fujisaki model, F0 contour on a logarithmic scale is assumed to be the superposition of three components: a phrase component, an accent component and a baseline component. The phrase component consists of the majorscale pitch variations over the duration of the prosodic units,
and the accent component consists of the smaller-scale pitch
variations in accented syllables. To avoid synthesizing oversmoothed and physically unlikely F0 contours, one reasonableapproach would be to incorporate the Fujisaki model into thestatistical model for speech synthesis so that we can separately
take the average of each of these components according to the

Copyright © 2014 ISCA

2. Generative model of speech F0 contours
2.1. Original Fujisaki Model
The Fujisaki model [1] assumes that an F0 contour on a logarithmic scale, y(t), where t is time, is the superposition of threecomponents: a phrase component yp (t), an accent componentya (t), and a base component yb :y(t) =yp (t) + ya (t) + yb .

(1)

The phrase component yp (t) consists of the major-scale pitch
variations over the duration of the prosodic units, and the accent component ya (t) consists of the smaller-scale pitch variations in accentual phrases. These two components are modeled
as the outputs of second-order critically damped filters, one being excited with a command function up (t) consisting of Diracdeltas (phrase commands), and the other with ua (t) consistingof rectangular pulses (accent commands):
yp (t) =Gp (t) ∗ up (t),{ 2 −αt
α teGp (t) =0ya (t) =Ga (t) ∗ ua (t),{ 2 −βt
β te
Ga (t) =
0

2322

(2)
(t ≥ 0),(t < 0)(3)(4)(t ≥ 0),(t < 0)(5)

14- 18 September 2014, Singapore

ters, Gp [k] and Ga [k], to generate the phrase and accent components, yp [k] and ya [k]:yp [k] = up [k] ∗ Gp [k],ya [k] = ua [k] ∗ Ga [k],

(7)
(8)

where ∗ denotes convolution over k. An F0 contour is thengiven byy[k] = yp [k] + ya [k] + yb ,where yb denotes the baseline value.

Figure 1: Previous HMM topology for command function modeling.

3. Context Dependent
Generative Model of F0 Contours

where ∗ denotes convolution over time. The baseline component yb is a constant value related to the lower bound of the
speaker’s F0 , below which no regular vocal fold vibration canbe maintained. α and β are natural angular frequencies of thetwo second-order systems, which are known to be almost constant within an utterance as well as across utterances for a particular speaker. It has been shown that α = 3 [rad/s] and β = 20
[rad/s] can be used as default values.
It is interesting to note that the phrase and accent commands, which we will henceforth refer to as the Fujisaki-model
parameters, can be interpreted as quantities related to linguistic
information. In the Japanese language, a phrase command andan accent command typically occur at the beginning of eachbreath group and over the range of accent nucleus in each accentual phrase, respectively.

3.1. Context dependent phrase and accent command
As discussed in Sec. 2.1, phrase and accent command is closely
associated with linguistic information such as breath group or
accent nucleus. Thus we assume that we can obtain an appropriate F0 contour from a text input by allocating a phrase command to the beggining of each breath group and an accent command to the range of each accent nucleus. Here, the problem is
how to determine the magnitudes of the phrase and accent commands. In this paper, we treat the magnitude of each command
as a model parameter to be trained using training data. We canassume that commands depend on the types of the preceding,current and succeeding breath groups and accentual phrases because of the fact that F0 contours represent the intonation ofnatural speech. Hence, the phrase and accent commands should
be determined according to the types of the preceding, current
and succeeding breath groups and accentual phrases. We henceforth call those environments context. In Sec. 3, we propose adecision-tree based context clustering algorithm, which allows
us to train the decision tree along with the model parameters
using the context labels.
In this paper, the following contextual factors are taken into
account:

2.2. Probabilistic formulation of F0 contour model
Here, we briefly review our probabilistic pitch contour model
based on the discrete-time version of the Fujisaki model [7, 8].
In the original Fujisaki model, the phrase commands andaccent commands are assumed to consist of Dirac deltas andrectangular pulses, respectively. In addition, they are not allowed to overlap each other. To incorporate these requirements,
we find it convenient to model the up [k] and ua [k] pair, i.e.,o[k] = (up [k], ua [k])T , using a hidden Markov model (HMM).
In [7, 8], we have assumed that {o[k]}K
k=1 is a sequence of
outputs generated from an HMM with the specific topology illustrated in Fig. 1. In state r0 , µp [k] and µa [k] are both constrained to be zero. In state p1 , referred to as the “phrase state,”µp [k] can take a non-zero value, Ap [k], whereas µa [k] is still
restricted to zero. At the phrase state, no selftransitions are allowed. In state r1 , µp [k] and µa [k] become zero again. This
path constraint restricts µp [k] to consisting of isolated deltas.
State r1 leads to states a1 , . . . , aN , referred to as the “accentstates.” At each accent state, µa [k] can take a different non-zero(n)value Aa , whereas µp [k] is forced to be zero. A direct statetransition from an accent state to a different state without passing through state r1 is not allowed. This path constraint restrictsµa [k] to consisting of rectangular pulses. The output distribution of each state is assumed to be a Gaussian distribution
o[k] ∼N (o[k]; csk , Υsk ) ,(9)• Contextual factors that are relevant to accent commands– mora count and accent type of {preceeding, current, suceeding} accentual phrase– position of current accentual phrase in current
breath group– position of current accentual phrase in sentence– {preceeding, current, suceeding} mora count ofbreath group– position of current breath group in sentence– mora count in sentence• Contextual factors that are relevant to phrase commands(6)– {preceeding, current, suceeding} mora count ofbreath group– accentual phrase count in {preceeding, current,suceeding} breath group– position of current breath group in sentence– mora count in sentence

where sk indicates the state variable. Namely, the mean vectorµ[k] = (µp [k], µa [k])T = csk and covariance matrix Σ[k] =Υsk are considered to evolve in time as a result of the statetransition s1 , . . . , sK . The definition of the above HMM can besummarized as follows:
Output sequence: {o[k]}K
k=1
State sequence: {sk }K
k=1
Output distribution: P (o[k]|sk ) = N (o[k]; csk , Υsk )Mean sequence: µ[k] = (µp [k], µa [k])T = csk
Transition probability: ϕi′ ,i = log P (sk = i|sk−1 = i′ )3.2. Probabilistic F0 contour model including
context dependent phrase and accent command with HMM
To extend the model presented in Subsec.
2.2 to acontex-dependent one, we consider the HMM describedin Fig.
2.
In this HMM, we would like the phrase
and accent states to be grouped into clusters via context clustering. We use M and N to denote the numbers of the phrase states and accent states, respectively.
Thus, the present HMM can be summarized as follows:

Given the state sequence s = {sk }K
k=1 , the above HMM
generates the up [k] and ua [k] pair. From (2) and (4), up [k]and ua [k] are then fed through different critically damped fil-2323

4. Parameter Training
and Generation Processes
4.1. Context clustering
In this section we propose an algorithm for training model pa(m)(n) Nrameters θ = {{Ap }M
m=1 , {Aa }n=1 } based on decision
tree context clustering [11] and the expectation-maximization(EM) algorithm [7, 8]. This algorithm allows us to train model
parameters using the training data and to generate phrase andcommand functions from an input sentence. In the following,
we will describe the case where the minimum description length(MDL) criterion is used for selecting nodes to be split. We
also select the probability density function of Fujisaki model
parameters θ and state sequence s given F0 contours y as thelikelihood in the MDL criterion. Here, the number of the leaf
nodes of the decision tree about phrase commands is equal to
the number of the phrase states pm , i.e., M . The number ofthe leaf nodes of the decision tree about accent commands isalso equal to the number of the accent states an in the present
HMM, i.e., N . Let us define d = 1, . . . , D as the index of thesentence in the training data, the F0 contour of d-th sentence as(d)y (d) = {y (d) [k]}K
k=1 and θ as the model parameters. Then the
MDL is given by

Figure 2: Proposed command function generative HMM. Unlike inthe case of previous HMM, command function is output from HMM in
which states are devided by context clustering.

Output sequence: o[k] = (up [k], ua [k])T (k = 1, . . . , K)
Set of states: S = {p0 , · · · , pM , a0 , · · · , aN }State sequence: s = {sk ∈ S|k = 1, . . . , K}
OutputP (o[k]|sk = i) = N (ci [k], Υ)(distribution:
)T

[ 2
]
(i ∈ p0 , a0 )(0, 0
)T
υp,i
0
(m)
ci = Ap , 0 (i ∈ pm ) Υ =
2
0
υa,i

(
(n) )T0, Aa(i ∈ an )Mean sequence: µ[k] = (µp [k], µa [k])T = csk

M DL = −L(θ) + c(N + M ) log W + C,
D {
∑
K (d)
1
log |Σ−1 | −log 2πL(θ) =
2
2
d=1

}
(11)
1 (d)
(y − µ(d) )T Σ−1 (y (d) − µ(d) ) ,2−1 (d)µ(d) = A−1 µ(d)µa + µb 1,
p +B( T )−1( )−1−1
Σ = A Σp A
+ B −1 Σa B T+ Σb ,−At the training stage, the positions of the breath groups andthe accent nucleus are assumed to be specified according to thehand-annotated context labels. Namely, the state sequence s is
assumed to be given and fixed during the training phase. Now,
let us define

where L(θ) denotes the log-likelihood function defined by(10),c denotes the weighting factor for the adjusting model size,
and C denotes the code length required for choosing the model,
which is assumed to be constant. Now we have to reestimate themodel parameter θ each time a node is split during the context
clustering process. An efficient parameter estimation algorithm
under a fixed model structure has already been proposed in [7].
Note that the proposed method is different from the previous
one in that the state sequence s is fixed and that the model parameters θ is defined differently. The overview of the proposed
method is shown in Fig. 3

K
y = {y[k]}K
k=1 , s = {sk }k=1 ,
(m) M
(n) N
o = {(up [k], ua [k])T }K
k=1 , θ = {{Ap }m=1 , {Aa }n=1 },Kµp= {µp [k]}K
k=1 , µa= {µa [k]}k=1 .

For simplicity, we treat µb , σp2 , σa2 , σb2 , α, β as constants.
(m)
(n) N
Here, θ = {{Ap }M
m=1 , {Aa }n=1 } are the free parameters
to be trained. In the same way as [7], the likelihood function of
the Fujisaki model parameters θ given y can be derived as{}|Σ−1 |1/2
1
T −1
P (y|θ) =
exp
−
(y
−
µ)
Σ
(y
−
µ)
,
2
(2π)K/2µ = A−1 µp + B −1 µa + µb 1,(10)( T )−1( T )−1−1−1
Σ = A Σp A
+ B Σa B+ Σb ,
whereΣp = σp2 I, Σa = σa2 I, Σb = σb2 I,
a
b
O
O
0
0
b1 b0
a1 a0

a a ab2 b1 b0

A= 2 .1 .0 .
,, B =  . . .
.. .. ..
.. .. ..
Oa2 a1 a0O
b2 b1 b0

Figure 3: Overview of the proposed context clustering algorithm.
4.2. Parameter training algorithm
Here we present a parameter inference algorithm that searches
for the unknown model parameter θ by iteratively updating θ
so as to maximize a lower bound function of the log-likelihood∑(d)|θ). For simplicity, we will hereafter omit thed log P (ysuperscript d in y (d) . By regarding x = (y Tp , y Ta , y Tb )T as thecomplete data this problem can be viewed as an incomplete data

a2 = (ψ − 1)2 , a1 = −2ψ(ψ − 1), a0 = ψ 2 ,b2 = (φ − 1)2 , b1 = −2φ(φ − 1), b0 = φ2 ,
1
1
ψ =1+
,φ = 1 +
.
αt0
βt0

2324

5.4

proposed
real speechlog F0

5.2

problem, which can be dealt with using the EM algorithm. The
log-likelihood function of θ given x is written as

5
4.8
4.6

1
1
log P (x|θ) = log |Λ−1 | − (x − m)T Λ−1 (x − m),
2
2


[ ]
yp
A−1 µpx = y a , m = B −1 µa  ,
(12)
yb
µb 1
 T −1A Σp A
O
O
−1
T
−1
Λ =
O
B Σa B
O .
O
O
Σ−1
b

0

c

c

1

1.5

2

2.5

3

3.5

4

4.5

Time [sec]proposedconventionallog F0

5.2
5
4.8
4.6
0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

Time [sec]Figure 4: An example of generated F0 contours of a japanese utterance “ippyo no kakusa wa sarani hirogaru darou”. The top graphshows the F0 contour generated by the simplified version of the present
method described in 5.1 along with the F0 contour of real speech. The
bottom graph shows the F0 contour generated by HTS [2].

In this case the Q function is thus given byQ(θ, θ ′ ) =

0.5

5.4

1[
log |Λ−1 | − tr(Λ−1 E[xxT |y; θ ′ ])2]
+ 2mT Λ−1 E[x|y; θ ′ ] − mT Λ−1 m . (13)according to the following criterion:
In above equation, a prior probability Pr(θ) is constant since θis uniformity distributed and the state sequence s is fixed.
M DL =

E[x|y; θ] and E[xxT |y; θ] are given explicitly as
E[x|y; θ] = m + ΛH T (HΛH T )−1 (y − Hm),

where xi indecates the magnitude of each command, J indicates the number of nodes in each tree, j indicates node index,
and Dj indicates the state occupancy of j-th node. We used the
HTS label sequence in [9] in order to extract the beginning ofeach breath group and the range of each accentual phrase. In
the method [8], the constant parameters were fixed respectively
at t0 = 8 ms, α = 3.0 rad/s, β = 20.0 rad/s, υn2 [k] = 1015 forunvoiced regions and υn2 [k] = 0.22 for voiced regions. µb wasset at the minimum log F0 value in the voiced regions.

E[xxT |y; θ] = Λ − ΛH T (HΛH T )−1 HΛ+ E[x|y; θ]E[x|y; θ]T ,by using the relationship y = Hx, where H = [I, I, I].
These are the values to be updated at the E step. Let E[x|y; θ]be partitioned into four K × 1 blocks such that E[x|y; θ] =(x̄Tp , x̄Ta , x̄Tb )T .
In the M step, There is no need to update the state sequence parameter s since we set restriction that it is always
fixed. Hence the M step update is equivalent to the procedure(m)(n)
that maximize with respect to Ap and Aa and is as follows.
∑
1
A(m)
=
[Ax̄p ]k , Tpm = {k|sk = pm },p|Tpm |

5.2. Experimental results
Fig. 4 shows an example of the F0 contours generated using
the last 53 sentences in the speech database available at [9], theraw F0 contours of real speech extracted with the straight analysis [10] and the F0 contours generated by HTS [2]. Since theconventional method is not ensured to generate an F0 contourthat follows the generating process of F0 contours, it tended tosometimes generate unnatural-sounding F0 contours. On theother hand, most of the F0 contours generated by the present
method sounded reasonably natural. This may be due to the factthat the Fujisaki model is able to express human F0 contours
consistently well. Some examples of the synthesized speech
generated by the present and conventional methods are demonstrated in our demo site {http://hil.t.u-tokyo.ac.
jp/˜kadowaki/Demos.htm}. We can also confirm fromthese results that the over-smoothing of F0 contours rarely occurred with the present approach.

k∈Tpm

A(n)
=
a

Dj {
}
(xi − µj )1∑
log(2πσj2 ) ++ cJ log W
2
σj2
d=1
(14)

∑
1
[B x̄a ]k , Tan = {k|sk = an }.
|Tan |k∈Tan

4.3. Parameter generation process
To obtain F0 contour from input text, we firstly extract the state
sequence by using context and constructed decision trees bytraining. Then, according to state sequence s and model parameters θ, we can obtain the F0 contour by determining y so
as to maximize P (y|s, θ) with respect to y.

6. Conclusion
This paper proposed a method for generating the F0 contourof speech from a text input for text-to-speech synthesis. We
previously introduced a statistical model describing the generating process of speech F0 contours, based on the discrete-timeversion of the Fujisaki model. To associate a sequence of theFujisaki-model parameters with a text input based on statistical learning, we extended this model to a context-dependentone. This idea was motivated by our expectation that averaging
these parameters would not directly cause the over-smoothingof the F0 contours, unlike the conventional method. We further
proposed a parameter training algorithm for the present model
based on a decision tree-based context clustering. The preliminary experimental results revealed that the present method wasable to generate natural-sounding F0 contours. Future work includes implementing the unified training algorithm described inSec. 4 and conducting subjective evaluations.

5. Preliminary experimentfor proposed method
5.1. Experimental conditions
As a preliminary experiment, we implemented a simplified version of the present method to demonstrate the proof of conceptof the present method. The simplified version we have implemented consists of two stages: First, we extract phrase and accent commands using the method described in [8] from the raw
F0 contours of the training data (the first 450 sentences of HTS
ver 2.1 demo script [9]). Second, the values of the extracted
phrase and accent commands were clustered using a decision
tree based context clustering. The decision tree was constructed2325

7. References[1] H. Fujisaki, In Vocal Physiology: Voice Production, Mechanisms
and Functions, Raven Press, 1988.
[2] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura, “Speech parameter generation algorithms for HMM-based
speech synthesis,” in Proc. ICASSP, vol. 3, pp. 1315–1318, 2000.
[3] H. Fujisaki and K. Hirose, “Analysis of voice fundamental frequency contours for declarative sentences of japanese,” J. Acoust.
Soc. Jpn (E), vol. 5, no. 4, pp. 233–242, 1984.
[4] H. Mixdorf, “A novel approach to the fully automatic extractionof fujisaki model parameters,” in Proc. ICASSP, 2000, vol. 3, pp.
1281–1284.
[5] S. Narusawa, N. Minematsu, K. Hirose, and H. Fujisaki, “A
method for automatic extraction of model parameters from fundamental frequency contours of speech,” in Proc. ICASSP, 2002,
pp. 509–512.
[6] Xu, Yi, and Santitham Prom-On, “Toward invariant functional representations of variable surface fundamental frequency contours:
Synthesizing speech melody via model-based stochastic learning,”
in Speech Communication 57, 2014, pp. 181–208.
[7] H. Kameoka, J. L. Roux, and Y. Ohishi, “A statistical model ofspeech F0 contours,” in Proc. SAPA, 2010, pp. 43–48.
[8] K. Yoshizato, H. Kameoka, D. Saito, and S. Sagayama, “Statistical approach to fujisaki-model parameter estimation from speech
signals and its quantitative evaluation,” in Proc. Speech Prosody
2012, 2012, pp. 175–178.
[9] “HMM-based Speech Synthesis System (HTS),” http://hts.
sp.nitech.ac.jp/[10] H. Kawahara, I. Masuda-Katsuse, and A. de Cheveigné, “Restructuring speech representations using a pitch-adaptive timefrequency smoothing and an instantaneous-frequency-based F0
extraction: Possible role of a repetitive structure in sounds,”
Speech Communication, vol. 27, no. 3, pp. 187–207, 1999.
[11] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, T. Kitamura,“ Simultaneous modeling of spectrum, pitch and duration in
HMM-based speech synthesis,” Proc. of Eurospeech, pp. 2347–
2350, 1999.

2326