INTERSPEECH 2014

Direct F0 Control of an Electrolarynx based on
Statistical Excitation Feature Prediction and its Evaluation through Simulation
Kou Tanaka, Tomoki Toda, Graham Neubig, Sakriani Sakti, Satoshi Nakamura
Graduate School of Information Science, Nara Institute of Science and Technology, Japan{ko-t, tomoki, ssakti, neubig, s-nakamura}@is.naist.jp

Abstract
An electrolarynx is a device that artiﬁcially generates excitation sounds to enable laryngectomees to produce electrolaryngeal (EL) speech. Although proﬁcient laryngectomees can produce quite intelligible EL speech, it sounds very unnatural dueto the mechanical excitation produced by the device. To address
this issue, we have proposed several EL speech enhancement
methods using statistical voice conversion and showed that statistical prediction of excitation parameters, such as F0 patterns,
was essential to signiﬁcantly improve naturalness of EL speech.
In these methods, the original EL speech is recorded with a microphone and the enhanced EL speech is presented from a loudspeaker in real time. This framework is effective for telecommunication but it is not suitable to face-to-face conversation because both the original EL speech and the enhanced EL speech
are presented to listeners. In this paper, we propose direct F0control of the electrolarynx based on statistical excitation prediction to develop an EL speech enhancement technique also
effective for face-to-face conversation. F0 patterns of excitation signals produced by the electrolarynx are predicted in realtime from the EL speech produced by the laryngectomee’s articulation of the excitation signals with previously predicted F0
values. A simulation experiment is conducted to evaluate theeffectiveness of the proposed method. The experimental results demonstrate that the proposed method yields signiﬁcant
improvements in naturalness of EL speech while keeping its intelligibility high enough.
Index Terms: laryngectomee, electrolarynx, electrolaryngeal
speech, statistical excitation prediction, simulation evaluation

Nasal cavity
Oral cavity

Vocal folds
Trachea

Tracheostoma

Electrolarynx
Expired air

Expired air

Figure 1: Speech production mechanisms of non-disabled people (left ﬁgure) and total laryngectomees (right ﬁgure).
noise causing degradation of sound quality. Naturalness is also
very low owing to the mechanical sound quality and artiﬁcial
fundamental frequency (F0 ) patterns caused by the mechanically generated excitation signals. In particular, the latter issue
is an essential drawback of EL speech caused by the difﬁcultyof artiﬁcially generating natural F0 patterns corresponding tolinguistic content.
To address these issues of EL speech, several EL speechenhancement methods have been proposed. These methods include enhancement methods based on a simple signal processing framework, e.g., noise reduction to alleviate the issues ofthe leaked excitation [1] and rule-based formant manipulationin analysis-synthesis [2]. Recently, statistical approaches to EL
speech enhancement have been proposed [3] to convert alaryngeal speech into target normal speech while keeping linguistic information unchanged. We recently proposed a hybrid approach [4] using noise reduction [1] [5] for enhancing spectral
parameters and statistical voice conversion [6] [7] for predicting
excitation parameters. Our experimental results demonstrated
that the hybrid approach achieved signiﬁcant improvements ofnaturalness while causing no degradation in intelligibility compared to the original EL speech. We have also found that theuse of F0 patterns statistically predicted from EL speech is very
effective for improving naturalness of EL speech.
Traditional EL speech enhancement systems need to record
the original EL speech with a microphone and present the enhanced EL speech with a loudspeaker. This enhancement process can be achieved in real time, and therefore it is very effective for facilitating human-to-human conversation. However,there is an essential drawback: both the original EL speech
and the enhanced EL speech can be heard in the vicinity of the
speaker. If these systems are used for telecommunication, no
problem is caused because it is possible to present only the enhanced speech to the listener. On the other hand, this technology
is not suitable for face-to-face conversation because the original
EL speech is always heard by the listener.
In this paper, we propose an EL speech enhancement system effective for any situation, including face-to-face conver-

1. Introduction
Speech is one of the most common media of human communication. Unfortunately, there are many people with disabilities that prevent them from producing speech freely, leading
to communication barriers. One example of people who cannot produce speech freely are laryngectomees, who have undergone an operation to remove the larynx including the vocal folds
for reasons such as an accident or laryngeal cancer. Laryngectomees cannot produce speech in the usual manner because theyno longer have their vocal folds.
Electrolaryngeal (EL) speech is produced by one of the
major alternative speaking methods for laryngectomees. As
shown in Figure 1, EL speech is produced using an electrolarynx, which is an electromechanical vibrator that is typicallyheld against the neck to mechanically generate artiﬁcial excitation signals. The generated excitation signals are conducted into
the speaker’s oral cavity, and EL speech is produced by articulating the conducted excitation signals. Compared with othertypes of alaryngeal speech, EL speech is relatively intelligible.
However, the excitation sounds are usually emitted outside as

Copyright © 2014 ISCA

Excitation sounds
leaked outside31

14- 18 September 2014, Singapore

sation. F0 patterns of the excitation signals produced by theelectrolarynx are directly controlled using statistical excitation
prediction. Namely, an F0 value at a current frame is predicted
in real time from the EL speech produced by the laryngectomee
articulating the excitation signals with previously predicted F0
values. Consequently, the proposed system has the potential toallow laryngectomees to directly produce enhanced EL speechwith more natural F0 patterns than the original EL speech, andpresent only the enhanced EL speech to the listener. As theﬁrst step toward implementation of the proposed system, a simulation experiment is conducted in this paper to demonstratethat the proposed system is capable of achieving signiﬁcant improvements in naturalness of EL speech while preserving its
high intelligibility.

  '   

  
& %       
       
%  &  

  

    

 !) 

2. Statistical Excitation Prediction 

      ( 

The proposed method uses a statistical voice conversion techniques [8] [9] to predict F0 patterns of normal speech produced
by a non-disabled person from spectral parameters of EL speech
produced by a laryngectomee. It consists of training and prediction processes as shown in Figure 2. A conversion model
to predict F0 patterns is trained in advance using a parallel
data set consisting of utterance pairs of EL speech by a laryngectomee and normal speech by a target non-disabled speaker.
The prediction process is based on maximum likelihood estimation of speech parameter trajectories considering global variance (GV) [9].

 

!  
 "#$       

 !






Figure 2: The training and prediction process.
(X,Y )

consists of source and target covaricovariance matrix Σm(XX)(Y Y )
ance matrices Σmand Σmand cross-covariance matri(XY )(Y X)
ces Σmand Σm . We also train a Gaussian distribution
modeling the probability density of the GV for F0 patterns of
the target normal speech [9].

2.1. Training Process

2.2. Prediction Process
The continuous F0 patterns of the target normal speech are predicted from the spectral segment features of EL speech using
the trained GMM as follows:

In the training process, ﬁrst source and target features are extracted from the parallel data. As the source features, spectral
segment features of EL speech are extracted from mel-cepstra at
multiple frames around the current frame [10]. As the target features, F0 values are extracted from natural speech. Continuous
F0 patterns [11] are generated from the originally extracted F0
values by spline interpolation to produce F0 values at unvoiced
frames, and low-pass ﬁltering is used to remove micro-prosody.
The effectiveness of using continuous F0 patterns in statisticalexcitation prediction was reported in [12].
We assume the spectral segment features of EL speech X t
and a log-scaled F0 value yt of normal speech at frame t. Asan output feature vector, we use Y t = [yt , Δyt ] consisting of
the static and dynamic features. We train a Gaussian mixture
model (GMM) to model the joint probability density [13] of thesource and target features using the corresponding joint feature
vector set generated by performing automatic frame alignment
for the parallel data set, which is given by:ŷ = argmax P (Y |X, λ)P (v(y)|λ(v) )ω
ysubject to Y = W y

P (X t , Y t |λ)(X,Y ) (X,Y )(1), Σm= M
m=1 αm N [X t , Y t ] ; μm
 (X) 
(XX)
(XY )
μm
Σm
Σm
(X,Y )
)
μ(X,Y
=
=
(2)
, Σm
m
(Y )
(Y X)
(Y Y )
μm
Σm
Σm

(3)


 
Y
=
where X=[X 
1 , · · · , Xt , · · · , XT ] , [Y 1 , · · · , Y t , · · · , Y T ] , and ŷ = [ŷ1 , · · · , ŷt , · · · , ŷT ]
are time sequence vectors of the spectral segment features, thejoint static and dynamic target F0 features, and the predicted
F0 features over an utterance, respectively. The matrix W isa transform to extend the static feature vector sequence into
the joint static and dynamic feature vector sequence [14]. The
GV probability density is given by P (v(y)|λ(v) ), where v(y)is the GV of the target static feature vector sequence y andλ(v) is a parameter set of the Gaussian distribution for the
GV. The GV likelihood weight is given by ω. Finally, silenceframes are automatically detected using waveform power andunvoiced excitation signals are generated only at those frames.
Note that a real time prediction process can be achieved byusing a computationally efﬁcient real-time voice conversion
method [15] based on the low-delay conversion algorithm toapproximately solve Eq. (3) [16].

3. Statistical Method for Directly
Controlling F0 Patterns of Electrolarynx

where  denotes transposition. N (·; μ, Σ) denotes a Gaussian
distribution with a mean vector μ and a covariance matrix Σ.
The mixture component index is m. The total number of mixture components is M . The parameter set of the GMM is λ,which consists of mixture-component weights αm , mean vec(X,Y )(X,Y )
tors μm
and full covariance matrices Σmfor individual)
consists of a
mixture components. The mean vector μ(X,Ym(Y )source mean vector μ(X)m and a target mean vector μm . The

3.1. Proposed System
The process of our proposed system is shown in Figure 3. Thissystem allows laryngectomees to directly produce enhanced EL
speech, and consists of two main processes: an articulation process and a prediction process. In the articulation process, theexcitation signals generated from the electrolarynx are articu-

32

Proposed system

lated by laryngectomees in the same manner as the traditional
speaking method using the electrolarynx. In the prediction process, F0 patterns of the excitation signals are predicted from EL
speech based on the real time statistical excitation prediction.
In this process, there are two main problems. One ismisalignment between the articulated sounds and F0 patterns.
Real-time statistical excitation prediction causes a constant processing delay of 50 to 70 msec as reported in [15]. Namely,
predicted F0 values from instances 70 msec before are used togenerate the excitation signals at the next frame. Consequently,
the EL speech produced by the proposed system always suffersfrom misalignment between the articulated sounds and F0 patterns caused by this delay. It is necessary to investigate whether
or not this misalignment causes perceivable degradation in the
EL speech.
The other problem is acoustic mismatches between the
training and prediction processes. The EL speech produced
by using the proposed system is affected by the predicted F0
values. Therefore, spectral parameters extracted from it arealso affected by them. This has the potential to cause acoustic
mismatches between the training and prediction processes. In
statistical excitation prediction, spectral analysis based on fast
Fourier transform (FFT) is usually used to signiﬁcantly reduce
computational cost. Because the FFT-based spectral analysis
easily captures periodicity of the excitation signals, the source
features (i.e., the mel-cepstral segment features) are strongly affected by the predicted F0 values. To address this issue, weinvestigate two approaches, a model-based approach and a feature extraction approach. The former approach uses the conversion model widely accepting EL speech with various F0 values. For the original EL speech samples in the parallel dataset, analysis-resynthesized EL speech samples are generated bymodifying the F0 values. FFT spectral features are extracted
from these generated samples and the resulting source features
are used in the GMM training. In this paper, global linear transformation is used for modifying the F0 . On the other hand, thelatter approach uses a spectral analysis method robust to periodicity of the excitation signals. STRAIGHT analysis [17] isused in this paper. To signiﬁcantly reduce computational costof STRAIGHT analysis, the predicted F0 value is directly usedin spectral analysis to avoid the F0 extraction process.

Simulation process
Electrolaryngeal speech

Articulation
Featureextraction

Spectralfeatures

F0 prediction

SpectralAperiodic
segment
componentsfeatures

Electrolaryngeal
speechPrediction with
GMM
Featureextraction
Processing
delay
Spectralsegment
featureDelayed F0
Mixed
excitation
generation

Articulation

Real-time
prediction
with GMM(w/ delay)

Delayed F0

Excitation signal

Featureextraction

Synthesis

Electrolaryngeal speech

Figure 3: The proposed system and its simulation implementation.
excitation model [19]. 5) Finally, the enhanced EL speech isapproximately synthesized by ﬁltering the generated excitation
signals with the extracted spectral envelope parameters. Notethat this is a result of using the spectral segment features extracted from the original EL speech, and therefore it is not affected by the predicted F0 patterns. To consider the impact ofthe predicted F0 patterns on the spectral segment features, 6)
the spectral segment features are extracted again from the synthesized EL speech and F0 pattern prediction is also performed
again using the extracted spectral segment features. Step 3) tostep 6) are iteratively repeated until the predicted F0 patterns
converge. If they converge, the proposed system may be expected to work stably because the EL speech produced with thepredicted F0 patterns is consistent with that used in the spectral segment feature extraction. We experimentally investigate
whether or not the predicted F0 patterns converge.

3.2. A Simulation Experient
As the ﬁrst step for implementation of the proposed system, weinvestigate the performance of the proposed system by a simulation experiment in this paper. The simulated implementation of
the proposed system is also shown in Figure 3. In the prediction
process, not the low-delay conversion algorithm but the batch
conversion algorithm is employed. The conversion accuracy of
the two algorithms is almost equivalent.
At ﬁrst, 1) we extract spectral envelope parameters and aperiodic components [18] from the original EL speech in advanceby using STRAIGHT analysis. These features capture acoustic properties depending on articulation and the excitation signals leaked out from the electrolarynx, except for periodicityof the excitation signals. These are used to approximate the
EL speech production process. Then, 2) spectral segment features are extracted from EL speech, and F0 patterns of normal
speech are predicted from them based on the statistical excitation prediction. 3) The predicted F0 patterns are delayed toconsider the delay time caused by real time prediction process.
4) Using the delayed F0 patterns and the extracted aperiodic
components, excitation signals are generated using the mixed

4. Experimental Evaluation
4.1. Experimental Conditions
We conducted an objective test for evaluating prediction accuracy of F0 patterns and two subjective evaluations on intelligibility and naturalness. The source speaker was one laryngectomee and the target speaker was one non-disabled speaker.
Both speakers recorded 50 phoneme-balanced sentences. Sampling frequency was set to 16 kHz.
We employed FFT analysis or STRAIGHT analysis to extract the spectrum parameters of EL speech. Note that F0 values of EL speech in STRAIGHT analysis were constantly set
to 100 Hz instead of performing STRAIGHT F0 analysis because F0 of the excitation signals was almost equivalent to 100
Hz in the electrolarynx used by the laryngectomee. The frame
shift length was set to 5 msec. The extracted spectral parameters were converted to the 0th through 24th mel-cepstral coefﬁcients, which were used to extract the mel-cepstral segment
feature as the source feature. The mel-cepstra at current ± 4
frames were used in this segment feature extraction. F0 values

33

5


!"
#$!%&$'#( ! ( ).1
!").1#$!%&$).1

Mean Opinion Score (MOS) 






4
3.5
3
2.5
2
1.5
1






  

EL

BASELINEMIXSTRAIGHT

Figure 5: Result of opinion test on intelligibility.



5
Mean Opinion Score (MOS)Confidence interval (95%)

4.5

Figure 4: Prediction accuracy for F0 correlation coefﬁcient.
of normal speech were extracted with STRAIGHT F0 analysis
and continuous F0 patterns were generated using a low-pass ﬁlter with 10 Hz cut-off frequency as the target feature. The mean
F0 value of normal speech was around 220 Hz.
We conducted a 5-fold cross validation test in which 40 utterance pairs were used for training, and the remaining 10 utterance pairs were used for evaluation. The number of mixture
components was set to 32. In the training data generation process described in Section 3.1, F0 values were shifted to 150,
200, and 250 Hz, and totally 160 EL speech samples were used
to train the GMM. The delay time in the simulation experiment
was set to 70 msec.
The EL speech generated by the following four systems
were mainly evaluated:

Confidence interval (95%)

4.5
4
3.5
3
2.5
2
1.5
1

EL

BASELINEMIXSTRAIGHT

Figure 6: Result of opinion test on naturalness.
“+matched”, the correlation coefﬁcient is constant over the iterative process in the simulation. On the other hand, it can beobserved from “NORMAL” that the correlation coefﬁcient signiﬁcantly degrades in the mismatched situations. This degradation is effectively alleviated by using the training data generation “MIX” or the robust spectral analysis “STRAIGHT.”
Figure 5 shows the result of the opinion test on intelligibility. “BASELINE” causes no degradation in intelligibility compared to the original EL speech as reported in [4]. In the proposed systems, “STRAIGHT” can also preserve high intelligibility of the original EL speech but “MIX” causes slight degradation in intelligibility. We found that F0 patterns generated in“MIX” sometimes varied unstably. Although we need to more
carefully analyze these results, it is possible that the number of
mixture components in “MIX” needs to be increased to accept
more varieties of the mel-cepstral segment features.
Figure 6 shows the result of the opinion test on naturalness.
The original EL speech is very unnatural but its naturalness canbe signiﬁcantly improved by “BASELINE” as reported in [4].
The proposed systems “MIX” and “STRAIGHT” can also signiﬁcantly improve the naturalness. Because no statistically signiﬁcant difference can be observed between “BASELINE” andthe proposed systems “MIX” and “STRAIGHT”, it is revealed
that misalignment of F0 patterns does not cause any degradationin naturalness.

EL Original EL speech
BASELINE Enhanced speech that does not perform real-time
F0 predictioin, and that has no processing delay causing
the F0 misalignment. This is equivalent to the conventional hybrid EL speech enhancement method [4] without the noise reduction process.
MIX Enhanced speech with the processing delay using the
GMM trained with the training data generation process.
STRAIGHT Enhanced speech with the processing delay using robust spectral analysis with STRAIGHT using thepredicted F0 .
In the objective evaluation, the correlation coefﬁcient between the predicted and natural F0 patterns was calculated. To
clarify the impact of the acoustic mismatches caused by thepredicted F0 on the F0 estimation accuracy, we also evaluated a system “NORMAL” with the processing delay using the
GMM without the training data generation process nor the robust spectral analysis. Moreover, the F0 estimation accuracy
not suffering from the acoustic mismatches was also evaluated in the systems, “MIX”, “STRAIGHT”, and “NORMAL”
by shifting the predicted F0 values so that their mean value
was equal to that of the original EL speech used in the training (i.e., 100 Hz), which were denoted as “MIX+matched”,“STRAIGHT+matched”, and “NORMAL+matched.”
In the subjective evaluation, we conducted two opinion tests
on intelligibility and naturalness. The opinion score was set toa 5-point scale (i.e., 1 (very poor) to 5 (excellent)). The numberof listeners was 5 in each test. Each listener evaluated naturalness and intelligibility of “EL”, “BASELINE”, “MIX”, and“STRAIGHT.”

5. Conclusions
In this paper, we proposed an electrolaryngeal (EL) speech enhancement system that directly controls F0 values of the excitation signals generated by an electrolarynx based on statisticalexcitation prediction. We conducted simulation experiments toevaluate the effectiveness of the proposed system, investigating
whether or not the enhanced EL speech is signiﬁcantly affectedby the processing delay of F0 prediction and acoustic mismatches caused by the dynamically predicted F0 values, which
are always observed in the proposed system. The experimental
results have shown that they cause no signiﬁcant differences ineither naturalness or intelligibility and the proposed system cansigniﬁcantly improve naturalness of EL speech while preserving its high intelligibility.

4.2. Experimental Results
Figure 4 shows the result of the objective evaluation. We cansee that correlation coefﬁcients of all systems converge and thesimulation process works reasonably well. If the acoustic mismatches are not caused by the predicted F0 , i.e., in the systems

6. Acknowledgements
Part of this work was supported by JSPS KAKENHI Grant
Number 26280060.

34

7. References[18] H. Kawahara, J. Estill, and O. Fujimura, “Aperiodicity extraction
and control using mixed mode excitation and group delay manipulation for a high quality speech analysis, modiﬁcation and synthesis system straight,” in Proc. MAVEBA, September 2001, pp.
13–15.

[1] H. Liu, Q. Zhao, M. Wan, and S. Wang, “Enhancement of electrolarynx speech based on auditory masking,” Biomedical Engineering, IEEE Transactions on, vol. 53, no. 5, pp. 865–874, May
2006.
[2] H. Sharifzadeh, I. McLoughlin, and F. Ahmadi, “Reconstructionof normal sounding speech for laryngectomy patients through a
modiﬁed CELP codec,” Biomedical Engineering, IEEE Transactions on, vol. 57, no. 10, pp. 2448–2458, October 2010.
[3] G. Aguilar-Torres, M. Nakano-Miyatake, and H. Perez-Meana,“Enhancement and restoration of alaryngeal speech signals,” inElectronics, Communications and Computers, 2006. CONIELECOMP 2006. 16th International Conference on, February 2006,
pp. 31–31.
[4] K. Tanaka, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, “Ahybrid approach to electrolaryngeal speech enhancement basedon spectral subtraction and statistical voice conversion.” in Proc.
INTERSPEECH, August 2013, pp. 3067–3071.
[5] S. Basha and P. Pandey, “Real-time enhancement of electrolaryngeal speech by spectral subtraction,” in Communications (NCC),
2012 National Conference on, February 2012, pp. 1–5.
[6] K. Nakamura, T. Toda, H. Saruwatari, and K. Shikano, “Speakingaid systems using gmm-based voice conversion for electrolaryngeal speech,” in Proc. Speech Communication, vol. 54, no. 1, January 2012, pp. 134 – 146.
[7] H. Doi, T. Toda, K. Nakamura, H. Saruwatari, and K. Shikano,“Alaryngeal speech enhancement based on one-to-many eigenvoice conversion,” Audio, Speech, and Language Processing,IEEE/ACM Transactions on, vol. 22, no. 1, pp. 172–183, January
2014.
[8] Y. Stylianou, O. Cappe, and E. Moulines, “Continuous probabilistic transform for voice conversion,” Speech and Audio Processing,IEEE Transactions on, vol. 6, no. 2, pp. 131–142, March 1998.
[9] T. Toda, A. Black, and K. Tokuda, “Voice conversion based on
maximum-likelihood estimation of spectral parameter trajectory,”
Audio, Speech, and Language Processing, IEEE Transactions on,
vol. 15, no. 8, pp. 2222–2235, November 2007.
[10] T. Toda, M. Nakagiri, and K. Shikano, “Statistical voice conversion techniques for body-conducted unvoiced speech enhancement,” Audio, Speech, and Language Processing, IEEE Transactions on, vol. 20, no. 9, pp. 2505–2517, November 2012.
[11] K. Yu and S. Young, “Continuous F0 modeling for HMM based
statistical parametric speech synthesis,” Audio, Speech, and Language Processing, IEEE Transactions on, vol. 19, no. 5, pp. 1071–
1079, July 2011.
[12] K. Tanaka, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, “Anevaluation of excitation feature prediction in a hybrid approachto electrolaryngeal speech enhancement,” in Proc. ICASSP, May
2014, pp. 4521–4525.
[13] A. Kain and M. Macon, “Spectral voice conversion for text-tospeech synthesis,” in Proc. ICASSP, vol. 1, May 1998, pp. 285–
288.
[14] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura, “Speech parameter generation algorithms for HMM-based
speech synthesis,” in Proc. ICASSP, vol. 3, June 2000, pp. 1315–
1318.
[15] T. Toda, T. Muramatsu, and H. Banno, “Implementation of computationally efﬁcient real-time voice conversion.” in Proc. INTERSPEECH, September 2012.
[16] T. Muramatsu, Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano,“Low-delay voice conversion based on maximum likelihood estimation of spectral parameter trajectory,” in Proc. INTERSPEECH,
September 2008, pp. 1076–1079.
[17] H. Kawahara, I. Masuda-Katsuse, and A. de Cheveigné, “Restructuring speech representations using a pitch-adaptive time–
frequency smoothing and an instantaneous-frequency-based F0
extraction: Possible role of a repetitive structure in sounds,” inProc. Speech Communication, vol. 27, no. 3. Elsevier, April
1999, pp. 187–207.

[19] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Maximum
likelihood voice conversion based on GMM with STRAIGHT
mixed excitation,” in Proc. INTERSPEECH, September 2006, pp.
2266–2269.

35