INTERSPEECH 2017
August 20–24, 2017, Stockholm, Sweden

Physically Constrained Statistical F0 Prediction
for Electrolaryngeal Speech Enhancement
Kou Tanaka1 , Hirokazu Kameoka2 , Tomoki Toda3 , and Satoshi Nakamura1
1

Graduate School of Information Science, Nara Institute of Science and Technology, Japan
2
NTT Communication Science Laboratories, NTT Corporation, Japan
3
Information Technology Center, Nagoya University, Japan
ko-t@is.naist.jp, kameoka.hirokazu@lab.ntt.co.jp, tomoki@icts.nagoya-u.ac.jp,
s-nakamura@is.naist.jp

Abstract

fundamental frequency (F0 ) pattern from the spectrum sequence of the EL speech based on Gaussian Mixture Models(GMMs) [1, 2, 3]. With a similar aim, whisper-to-speech conversion [4] and whisper-to-audible speech conversion [5] havebeen proposed. These methods have successfully shown to improve the naturalness of EL speech [1, 2] while preserving its
intelligibility [3]. However, the predicted F0 patterns are still
unnatural compared with that in normal speech. One possible reason is that the predicted F0 patterns were not necessarily
guaranteed to satisfy the physical constraints of the actual control mechanism of the thyroid cartilage, even though they were
optimal in a statistical sense.

Electrolaryngeal (EL) speech produced by a laryngectomee using an electrolarynx to mechanically generate artificial excitation sounds severely suffers from unnatural fundamental frequency (F0 ) patterns caused by monotonic excitation sounds.
To address this issue, we have previously proposed EL speechenhancement systems using statistical F0 pattern prediction
methods based on a Gaussian Mixture Model (GMM), making
it possible to predict the underlying F0 pattern of EL speechfrom its spectral feature sequence. Our previous work revealed
that the naturalness of the predicted F0 pattern can be improvedby incorporating a physically based generative model of F0
patterns into the GMM-based statistical F0 prediction system
within a Product-of-Expert framework. However, one drawback of this method is that it requires an iterative procedure
to obtain a predicted F0 pattern, making it difficult to realizea real-time system. In this paper, we propose yet another approach to physically based statistical F0 pattern prediction byusing a HMM-GMM framework. This approach is noteworthy
in that it allows to generate an F0 pattern that is both statisticallylikely and physically natural without iterative procedures. Experimental results demonstrated that the proposed method wascapable of generating F0 patterns more similar to those in normal speech than the conventional GMM-based method.
Index Terms: electrolaryngeal speech, statistical F0 prediction,
generative model, speech enhancement

As for the generative process of F0 patterns, one of theauthors have proposed a statistical model [7, 8, 9] formulated
as a stochastic counterpart of the Fujisaki model [10], a wellfounded mathematical model representing the control mechanism of vocal fold vibration. The Fujisaki model [10] assumes
that a F0 pattern on a logarithmic scale is the superposition of aphrase component, an accent component and a constant value.
The phrase and accent components are assumed to be associated with mutually independent types of movement of the thyroid cartilage with different degrees of freedom and muscular
reaction times. The model reported in [7, 8, 9] has made it
possible to estimate the underlying parameters of the Fujisaki
model that best explain the given F0 pattern, by using powerful statistical inference techniques. We previously proposed
unifying this model and the GMM-based F0 pattern prediction
model within a Product-of-Experts (PoE) framework [6]. Although this framework showed improvement in the prediction
accuracy, it turned out that it was not suitable to real-time processing because of the iterative procedure required in the prediction process.

1. Introduction
Speech is a common tool in human communication. Since
speech is produced by the vocal apparatus, the produced sounds
are physically constrained by the conditions of human body.
Unfortunately, there are many people with disabilities that prevent them from producing speech freely, leading to communication barriers and degrading Quality of Life (QoL). Laryngectomees are people with disabilities and have undergone an operation to remove the larynx including the vocal folds for several reasons such as injury and laryngeal cancer. Their ability togenerate excitation sounds is severely impaired, because they nolonger have their vocal folds although their vocal tracts remain.
One alternative means of producing speech sounds is the use ofelectrolaryngeal (EL) speech, which allows them to use the excitation sounds mechanically generated from an electrolarynx.
EL speech is reasonably intelligible, but somewhat unnatural
particularly due to the monotonic excitation sounds.
To address this issue, we have previously proposed methods that make it possible to convert acoustic features of EL
speech to those of normal-sounding speech by predicting the

Copyright © 2017 ISCA

In this paper, we propose yet another approach that makes
it possible to use the generative model of F0 patterns for statistical F0 pattern prediction using an HMM-GMM. Specifically,
we model the joint probability distribution of a sequence of thephrase/accent commands and a spectral feature sequence. As
we shall see in the following, the state sequence of our HMM
is associated with a sequence of the phrase/accent commandsof the Fujisaki model. Here, we assume that this HMM emitsa spectral feature vector at each state. By doing so, our HMM
can be used as a model to predict a phrase/accent command sequence from an input spectral feature sequence. With properly
trained parameters, the most likely phrase/accent command sequence given an input spectral feature sequence can be foundby state decoding. Experimental results in term of prediction
accuracy revealed that the proposed method was capable of predicting F0 patterns more similar to those in normal speech than1069

http://dx.doi.org/10.21437/Interspeech.2017-688

the conventional GMM-based method.

2. Statistical F0 Pattern Prediction
Based on GMM
We briefly review our statistical F0 prediction method [1, 2, 3],
based on statistical voice conversion techniques [11, 12]. The
aim of this method is to predict F0 patterns from the spectralfeatures of EL speech. As with voice conversion methods, itconsists of training and prediction processes.
Let λG be the parameters of the joint probability densityp((x[t]⊤ , o[t]⊤ )⊤ |λG ) defined as a GMM, where ⊤ is transposition, and x[t] and o[t] are a source feature and a target feature
at time frame t, respectively. The corresponding joint feature
vectors can be obtained by performing automatic frame alignment with Dynamic Time Warping. As a source feature, thespectral segment feature x[t] of EL speech is obtained by applying principal component analysis (PCA) to the stacked vector consisting of the mel-cepstra of multiple frames around thecurrent frame t [13]. The target feature o[t] = (y[t], ∆y[t])⊤consists of the static and delta (time derivative) components of
the log-scaled F0 value y[t], extracted on a frame-by-frame basis from the target normal speech. At training time, the parameters of the GMM are trained using the parallel data of source
and target features.
At test time, given the spectral segment sequence x =(x[1]⊤ , . . . , x[T ]⊤ )⊤ of EL speech, the most likely F0 sequence y = (y[1], . . . , y[T ])⊤ can be obtained as follows:ŷ = argmax p(o|x, λG ) subject to o = W y,

Figure 1: Original Fujisaki model [10].

(1)ywhere o = (o[1]⊤ , . . . , o[T ]⊤ )⊤ is the joint static and dynamic
feature vector sequence, and W is a constant matrix that transforms the static feature vector sequence y to o.

3. Generative Model of F0 Patterns
The generative model of F0 patterns [7, 8, 9] is a stochastic counterpart of a discrete-time version of the Fujisaki
model [10]. The Fujisaki model (shown in Fig. 1) assumes that
F0 patterns y[t] on a logarithmic scale is given as the sum of
three components:
y[t] = yp [t] + ya [t] + µb ,Figure 2: Command function modeling with HMM.

(2)

where yp [t] and ya [t] are a phrase component and an accent
component at time frame t, and µb is a constant value, respectively. The phrase and accent components are assumed to be theoutputs of different second-order critically damped filters, gp [t]and ga [t], excited with Dirac deltas up [t] (phrase commands)
and rectangular pulses ua [t] (accent commands), respectively:
yp [t] = gp [t] ∗ up [t],ya [t] = ga [t] ∗ ua [t],Figure 3: State splitting.

(3)
(4)

the magnitude of each phrase/accent command can take. It canthus be understood as the resolution of magnitude “quantization”. At states r0 and r1 , the means µp [t] = E[up [t]] andµa [t] = E[ua [t]] are both restricted to zero. At states p0 ,. . .,pM −1 , µp [t] can take non-zero values, whereas µa [t] is still restricted to zero. At states a0 ,. . ., aN −1 , µa [t] can take non-zerovalues, whereas µp [t] is forced to be zero. The path constraint
shown in Fig. 2 restricts µp [t] to consisting of isolated deltas
and µa [t] to consisting of rectangular pulses. Furthermore, we
split each state (except for pm ) into a certain number of substates (as in Fig. 3) so that we can directly assign probabilities
to the durations of self-transition.

where ∗ is convolution over time.
The key idea of the generative model proposed in [7, 8, 9] isthat a sequence of phrase/accent command pair is modeled as an
output sequence of the following path-restricted hidden Markov
model (HMM) with Gaussian emission densities (shown in
Fig. 2) so that estimating the state transition of the HMM directly amounts to estimating the Fujisaki-model parameters.
This HMM λF consists of M+N+2 states, r0 , r1 , p0 ,. . ., pM −1 ,a0 ,. . ., aN −1 . M and N are the numbers of possible values that

1070





transitions. This is in particular convenient since phrase/accentcommand sequences have several requirements to be met [10].
We then use the method in [7, 8, 9] to extract thephrase/accent commands from target speech and determine thestate sequence s. With fixed s, we can train the state emis⊤[ 2](i ∈ r0 , r1 )
 (0, 0)sion density of each state by using the spectral feature vectors
v
0
(m)
p,i
,
ci =
, Σi =(µp , 0)⊤ (i ∈ pm )2of source EL speech at all the frames assigned to that state. To
0
va,i

(n) ⊤
(0, µa )(i ∈ an )avoid overfitting, we consider reducing the parameters of theState transition probabilities: ϕî,i = p(s[t] = i|s[t − 1] = î).
HMM by parameter tying. Specifically, we partition the sub
 states in each state into three segments so that all the substates
belonging to the same segment have exactly the same emission
By using the conditional density p(y|u) reflecting the condensities. As for the spectral feature, we use the spectral segstraints Eq. (2,3,4) and the HMM likelihood p(u|s, λF ), we
ment feature used in the method described in Sec. 2. Namely,
can describe the probability density of y as:
the spectral segment feature x[t] is obtained by applying PCA∫
tothe stacked vector consisting of the mel-cepstra of multiplep(y|s, λF ) =p(y|u)p(u|s, λF ) du(5)
frames around the current frame t.
= N (y; m, Γ),(6)
4.3. Prediction Process
where m is given as the F0 pattern of the Fujisaki model given
Given the spectral feature sequence x of EL speech, we cana phrase/accent command sequence s [7, 8, 9].
predict the most likely state sequence ŝ by maximizing the following conditional probability:
Output sequence: u[t] = (up [t], ua [t])⊤ (t = 1, . . . , T ),Set of states: S = {r0 , p0 , . . . , pM −1 , r1 , a0 , . . . , aN −1 },State sequence: s ={s[t] ∈ S|t = 1, . . . , T },Emission
densities: p(u[t]|s[t] = i) = N (u[t]; ci , Σi ),

4. Proposed F0 Pattern Prediction
Based on HMM-GMMŝ = argmax p(s|x, λh )

(11)

s

4.1. Joint Modeling with HMM-GMM= argmax p(x, s|λh ).

p(x, s|λh ) = p(x|s, λh )p(s|λh ),wherep(s|λh ) =∏Given ŝ, we can obtain the F0 pattern ŷ by maximizing Eq. (5)with respect to y. Since p(y|s, λF ) is given as a multivariate
normal distribution [9], ŷ is simply given by its mean. In contrast with the conventional GMM-based approach, the present
framework always ensures the predicted F0 pattern to follow
the Fujisaki model. This restriction is perhaps too strong, but
may be advantageous in that it can effectively avoid generating
unnatural patterns.

5. Experimental Evaluation
5.1. Experimental Conditions
We objectively evaluated the performance of the proposed
method with the following F0 correlation coefficients fcorr between the predicted F0 patterns ŷ and target F0 patterns y atvoiced frames:(7)

fcorr =p(s[t]|s[t − 1], λh ),(8)p(x|s, λh ) =p(x[t]|s[t], λh ).

(9)t

We assume the state emission density to be a GMM:∑p(x[t]|s[t] = i) =αi,m N (x[t]; µi,m , Σi,m ).

2
σŷy
,
σŷ σy(13)2where σŷy
is a covariance of the predicted and the target F0
patterns, and σŷ and σy are standard deviations of the predicted
and the aliged reference F0 patterns, respectively. We chose theGMM-based method [1, 2, 3] as a baseline method for comparison. We also tested a simple postfiltering approach that consistsof performing the GMM-based F0 prediction followed by fitting the Fujisaki model to the predicted F0 pattern using themethod of [7, 8, 9].
For the training, we recorded EL speech uttered by one normal person (male) and normal speech uttered by one different
normal person (male). Each speaker uttered about 503 sentences in the ATR phonetically balanced sentence set [14]. We
used 450 sentences as training data, and remaining 53 sentences
as test data. The sampling frequency was set to 16 kHz. The
frame shift was set to 5 ms. The frame length of feature extraction was set to 25 ms. The dimension of mel-cepstra including
waveform power information was set to 25. The concatenated

t

∏

(12)

s

Here, we propose designing the joint probability distribution ofa sequence of the phrase/accent commands of target speech anda spectral feature sequence of source speech. The state sequenceof the HMM introduced in Sec. 3 is associated with a sequenceof the phrase/accent commands of the Fujisaki model. Thus, if
we assume that this HMM λh emits a spectral feature vector ateach state according to a GMM emission density, the joint probability of output and state sequences of this HMM-GMM can beused as a model to predict a phrase/accent command sequence
from an input spectral feature sequence. With properly trained
parameters, the most likely phrase/accent command sequence
given an input spectral feature sequence can be found by state
decoding.
We can write the joint p.d.f. of the spectral feature sequence
x of EL speech and the state sequence s of the phrase/accentcommands of normal speech as:

(10)

m

4.2. Training Process
As with the method in Sec. 2, we use the parallel data ofsource and target speech obtained using DTW for training.
First, we train the state transition probabilities of the pathrestricted HMM so that the HMM is ensured to produce only
the phrase/accent command sequences that are likely to occur innormal speech. By setting the transition probability of a particular pair of states at exactly 0, we can impose constraints on state1071

Frequency [Hz]frames were ±4 frames around current frame when we performthe PCA. Thus, we compressed a 25 ∗ (4 + 1 + 4) = 225 dimensions feature into a 50 dimensions feature. The extractionof mel-cepstra of EL speech uses FFT analysis. The extraction of mel-cepstra of normal speech uses STRAIGHT analysis [15]. The extraction of F0 of normal speech is performed
by TEMPO [15]. The level of amplitude of phrase command
and accent command were set to 3, respectively. The number of mixture components are 16 and 4 for the GMM-based
method and the proposed method, respectively. In this paper, toaddress the smaller amount of spectral features on phrase commands compared with that on accent commands, we modified
the Dirac deltas of phrase command to rectangular pulses of 5frames which amplitude is one-fifth amplitude of corresponding
Dirac delta.
The methods selected for comparison were:a) Target F0 patternsb) F0 patterns reconstructed from target F0 patterns

400
200
100
50

0

100

200

Magnitude

1.5
1.0

Frequency [Hz]Frequency [Hz]

0

100

200

300

400

500

600

100
0

100

200

300

400

500

600

700

800

f) F0 patterns predicted by proposed method

400
200
100
50

0

100

200

Magnitude

300

400

500

600

700

800

g) Phrase commands predicted by proposed methodh) Accent commands predicted by proposed method

1.0
0.5
0.0

0

100

200

300

400
Frame index

500

600

700

800

Figure 4: Sample of F0 patterns and phrase and accent commands.
This is because an intonation pattern is usually determined by
the grammatical structure of an uttered sentence or the accent
type associated with each phrase. Thus, it would be natural tohypothesize that all phrase and accent command sequences are
drawn from a vocabulary consisting of relatively small numberof intonation pattern templates. By using a dictionary consistingof a finite number of left-to-right HMM templates, we can assume that a sequence of the phrase and accent command pairs is
generated according to a concatenation of those templates. Thiscan be modeled by an HMM topology employed in [16].

5.3. Discussion
Our proposed method allows us to directly predict the phrase/accent commands from the spectral features. This allows us tomodify the F0 patterns within the physical constraint underlying the generative process of F0 patterns. It should be noted thatthe present model can also be developed into our previously proposed methods, such as [3] and [6], for further improvements.
While the GMM-based method allows us to predict raw F0
values, the phrase commands and accent commands are quantized into 3 levels in the proposed method, described in Sec. 3
and 5. The accuracy of F0 pattern prediction suffers from thisnegative impact of quantization.
To further improve the prediction accuracy, we also plan
to incorporate the model employed in [16]. In normal speech,
many phrases or sentences share the same intonation pattern.

6. Conclusions
In this paper, to improve F0 prediction performance in electrolaryngeal speech enhancement, we proposed a HMM-GMM
based F0 pattern prediction that combined two conventional
methods, a statistical F0 pattern prediction method and a statistical F0 pattern modeling method based on its generative process. Experimental results revealed that the proposed method
outperformed our conventional method in terms of prediction
accuracy.

Table 1: F0 correlation coefficients
gmm
gmm+post
proposed

600

200

1.5

Table 1 shows that using the postfiltering slightly improved
the prediction accuracy obtained with the GMM-based method.
This is because the predicted F0 patterns are modified so thatthe physical constraint expressed by the Fujisaki model is satisfied (see Fig. 4). We confirmed that the proposed method
yielded even higher accuracy than both of the baseline methods.
This implies the advantage of the proposed method, which allows for the direct prediction of the phrase/accent commands ofthe Fujisaki model. These show that modeled natural duration
and amplitude of F0 command help us to generate F0 patterns
from spectral features and the use of fixed transition probability
is effective.

500e) F0 patterns predicted by GMM-based method

400

50

5.2. Experimental Results

400

0.5
0.0

• gmm: the GMM-based maximum likelihood trajectory
estimation of F0 patterns by using the F0 feature of normal speech as the target feature.
• gmm+post: As the post processing, fit the Fujisaki modelto the above predicted F0 patterns,• proposed: Predict F0 patterns with the proposed HMMGMM based method.

300c) Phrase commands reconstructed from target F0 patternsd) Accent commands reconstructed from target F0 patterns

7. Acknowledgements

0.40
0.41
0.50

This work was supported in part by JSPS KAKENHI Grant
Number 26280060, and by JST, PRESTO Grant Number JPMJPR1657.

1072

8. References[1] K. Nakamura, T. Toda, H. Saruwatari, and K. Shikano,“Speaking-aid systems using gmm-based voice conversion forelectrolaryngeal speech,”in Proc. Speech Communication,
vol. 54, pp. 134– 146, January 2012.
[2] H. Doi, T. Toda, K. Nakamura, H. Saruwatari, and K. Shikano,“Alaryngeal speech enhancement based on one-to-many eigenvoice conversion,” Audio, Speech, and Language Processing,IEEE/ACM Transactionson, vol. 22, no. 1, pp. 172–183, January
2014.
[3] K. Tanaka, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, “Ahybrid approach to electrolaryngeal speech enhancement basedon noise reduction and statistical excitation generation,” IEICE
Transactions on Information and Systems, vol. E97-D, no. 6, pp.
1429–1437, June 2014.
[4] J. Li, I. V. McLoughlin, L. Dai, and Z. Ling, “Whisper-to-speech
conversion using restricted Boltzmann machine arrays,” Electronics Letters, vol. 50, no. 24, pp. 1781–1782, November 2014.
[5] M. Janke, M. Wand, T. Heistermann and T. Schultz, “Fundamental frequency generation for whisper-to-audible speech conversion,” Proc. ICASSP, pp. 2598–2602, May 2014.
[6] K. Tanaka, H. Kameoka, T. Toda, and S. Nakamura, “Statistical
F0 prediction for electrolaryngeal speech enhancement considering generative process of F0 contours within product of experts
framework,” Proc. ICASSP, pp. 5665–5669, March 2016.
[7] H. Kameoka, J. Le Roux, Y. Ohishi, “A statistical model of speech
F0 contours,” ISCA Tutorial and Research Workshop on Statistical And Perceptual Audition, pp. 43–48, September 2010.
[8] K. Yoshizato, H. Kameoka, D. Saito, and S. Sagayama, “Statistical approach to fujisaki-model parameter estimation from speech
signals and its quantitative evaluation,” Proc. Speech Prosody, pp.
175–178, May 2012.
[9] H. Kameoka, K. Yoshizato, T. Ishihara, K. Kadowaki, Y. Ohishi,
and K. Kashino, “Generative modeling of voice fundamental
frequency contours,” Audio, Speech, and Language Processing,IEEE/ACM Transactionson, vol. 23, no. 6, pp. 1042–1053, June
2015.
[10] H. Fujisaki, “A note on the physiological and physical basis for thephrase and accent components in the voice fundamental frequency
contour,” Vocal Fold Physiology: Voice Production, Mechanisms
and Functions, pp. 347–355, 1998.
[11] Y. Stylianou, O. Cappe, and E. Moulines, “Continuous probabilistic transform for voice conversion,” Speech and Audio Processing,IEEE Transactions on, vol. 6, no. 2, pp. 131–142, March 1998.
[12] T. Toda, A.W. Black, and K. Tokuda, “Voice conversion based on
maximum-likelihood estimation of spectral parameter trajectory,”
IEEE Transactions on Audio, Speech, and Language Processing,vol. 15, no. 8, pp. 2222–2235, November 2007.
[13] T. Toda, M. Nakagiri, and K. Shikano, “Statistical voice conversion techniques for body-conducted unvoiced speech enhancement,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 20, no. 9, pp. 2505–2517, November 2012.
[14] M. Abe, Y. Sagisaka, T. Umeda, and H. Kuwabara, “Speech
database,” ATR Technical Report, TR-I-0166, September 1990.
[15] H. Kawahara, I. Masuda-Katsuse, and A. Cheveigne, “Restructuring speech representations using a pitch-adaptive time-frequency
smoothing and an instantaneous-frequency-based F0 extraction:
Possible role of a repetitive structure in sounds,” in Proc. SpeechCommunication, Vol. 27, No. 3-4, pp. 187–207, April 1999.
[16] T. Ishihara, H. Kameoka, K. Yoshizato, D. Saito, S. Sagayama,“Probabilistic speech F0 contour model incorporating statistical
vocabulary model of phrase-accent command sequence,” in Proc.
Interspeech, pp. 1017–1021, August 2013.

1073