INTERSPEECH 2013

A Hybrid Approach to Electrolaryngeal Speech Enhancement Based on Spectral
Subtraction and Statistical Voice Conversion
Kou Tanaka, Tomoki Toda, Graham Neubig, Sakriani Sakti, Satoshi Nakamura
Graduate School of Information Science, Nara Institute of Science and Technology, Japan{ko-t, tomoki, ssakti, neubig, s-nakamura}@is.naist.jp

Abstract
We present a hybrid approach to improving naturalness ofelectrolaryngeal (EL) speech while minimizing degradation in intelligibility. An electrolarynx is a device that artificially generates excitation sounds to enable laryngectomees to produce EL
speech. Although proficient laryngectomees can produce quite intelligible EL speech, it sounds very unnatural due to the mechanical excitation produced by the device. Moreover, the excitation
sounds produced by the device often leak outside, adding noise to
EL speech. To address these issues, previous work has proposed
methods for EL speech enhancement through either noise reduction or voice conversion. The former usually causes no degradation in intelligibility but yields only small improvements in naturalness as the mechanical excitation sounds remain essentiallyunchanged. On the other hand, the latter method significantly
improves naturalness of EL speech using spectral and excitation
parameters of natural voices converted from acoustic parametersof EL speech, but it usually causes degradation in intelligibility
owing to errors in conversion. We propose a hybrid method using the noise reduction method for enhancing spectral parametersand voice conversion method for predicting excitation parameters. The experimental results demonstrate the proposed method
yields significant improvements in naturalness compared with EL
speech while keeping intelligibility high enough.
Index Terms: speaking-aid, electrolaryngeal speech, spectral
subtraction, voice conversion, hybrid approach

Figure 1: Speech production mechanisms of non-disabled people(left figure) and total laryngectomees (right figure).
relatively intelligible. However, there are also some issues of EL
speech: e.g., 1) the excitation sounds are usually emitted outside
as noise causing degradation of sound quality, and 2) naturalness
is low owing to its mechanical sound quality caused by the mechanically generated excitation signals. In particular, the latterissue is an essential drawback of EL speech caused by the difficulty of artificially generating natural F0 patterns corresponding
to linguistic content.
To address these issues of EL speech, two main approaches
have been proposed. One is based on noise reduction [1] andthe other is based on statistical voice conversion (VC) [2] [3].
The former approach aims to reduce the effect of the excitation
sounds leaked from the electrolarynx by using noise reduction
techniques, such as spectral subtraction (SS) [4]. This noise reduction process causes no degradation in intelligibility but yields
only small improvements in naturalness as the mechanical excitation sounds remain essentially unchanged. On the other hand, thelatter method is capable of significantly improving naturalness by
converting acoustic parameters of EL speech into those of natural
voices using statistical VC techniques [5] [6]. The use of statistics
extracted from a parallel data set consisting of EL speech and natural voices makes it possible to achieve more complex conversion
processes than that of other signal processing approaches, such asformant manipulation [7]. For example, it is possible to convert
from a spectral parameter sequence of EL speech into F0 patternsof natural voices. However, VC-based approaches usually cause
degradation in intelligibility owing to errors in conversion [3].
In this paper, to develop an EL speech enhancement method
for significantly improving naturalness while preserving intelligibility in EL speech, we propose a hybrid method using theSS-based noise reduction method for enhancing spectral parameters and the VC method for predicting excitation parameters.
Furthermore, to avoid degradation in intelligibility caused byunvoiced/voiced prediction errors, we also propose an estimation method of continuous F0 patterns. We conduct an experimental evaluation, which demonstrates that the proposed method

1. Introduction
Speech is one of the most common media of human communication. Unfortunately, there are many people with disabilities thatprevent them from producing speech freely, leading to communication barriers. One example of people who cannot produce
speech freely are laryngectomees, who have undergone an operation to remove the larynx including the vocal folds for reasons
such as an accident or laryngeal cancer. Larengectomees cannot
produce speech in the usual manner because they no longer havetheir vocal folds. Therefore, they require another method to produce speech without the vocal fold vibration.
Electrolaryngeal (EL) speech is produced by one of the majoralternative speaking methods for laryngectomees as shown in Figure 1. EL speech is produced using an electrolarynx, which is anelectromechanical vibrator that is typically held against the neckto mechanically generate artificial excitation signals. The generated excitation signals are conducted into the speaker’s oral cavity,and EL speech is produced by articulating the conducted excitation signals. There are several advantages of EL speech comparedwith other types of alaryngeal speech, such as esophageal speech:
e.g., 1) it is easy to learn how to produce EL speech, 2) less physical power is needed to produce EL speech, and 3) EL speech is

Copyright © 2013 ISCA

3067

25- 29 August 2013, Lyon, France

yields significant improvements in naturalness compared with EL
speech while causing no degradation in intelligibility.

likelihood estimation of speech parameter trajectories considering global variance (GV) [6]. This framework is the same asin conversion from body-conducted unvoiced speech into normal
speech [10].

2. Electrolaryngeal Speech Enhancement
Based on Spectral Subtraction (SS)3.1. Training Process

SS is a method for restoration of the amplitude spectrum of aspeech signal that has been observed with additive noise. Thisis done through subtraction of an estimate of the amplitude spectrum of the noise from the amplitude spectrum of the noisy speech
signal. The noisy speech signal model in the frequency domain isexpressed as follows:
Y (ω, t) = S(ω, t) + L(ω, t)(1)

where Y (ω, t), S(ω, t), and L(ω, t) are respectively componentsof the noisy speech signal, the clean speech signal, and the additive noise signal at frequency ω and time frame t. Assumingthat the additive noise signal is stationary, the generalized SS
scheme [8] is described as follows:{γ
1|Y (ω, t)|γ − α|L̂(ω)|γ ( |Y|L̂(ω)|γ < α+β )γ(ω,t)||Ŝ(ω, t)| =(2)γβ|L̂(ω)|(otherwise)
where α (α > 1) is an over-subtraction parameter, β (0 ≤ β ≤1) is a spectral flooring parameter, γ is an exponential domain parameter, and L̂(ω) is an estimate of the averaged amplitude spectrum of the additive noise signal.
In this paper, we implement SS for EL speech enhancement,
as shown in Figure 2. The averaged amplitude spectrum of the additive noise signal is estimated in advance using the excitation signals generated from the electrolarynx. In order to record only the
excitation signals leaked from the electrolarynx as accurately as
possible, the excitation signals are recorded with a close-talking
microphone while keeping speaker’s mouth closed. The excitation signals are generated with the electrolarynx held in the usual
manner, as shown in Figure 1.

Let us assume the spectral segment features of EL speech X t
and a static feature vector y t of each type of the normal speechparameters at frame t. As an output speech feature vector, we use⊤ ⊤Y t = [y ⊤t , ∆y t ] consisting of the static and dynamic features,
where ⊤ denotes transposition of the vector. We independently
train three GMMs to model the joint probability densities [11] of
the spectral segment feature of EL speech and each of the output
feature vectors of individual target parameters of normal speechusing the corresponding joint feature vector set as follows:
P (X t , Y t |λ)()∑(X,Y )⊤⊤ ⊤(X,Y )= M, Σm(3)
m=1 αm N [X t , Y t ] ; µm[]
[ (X) ]
(XX)
(XY )
µm
Σm
Σm
(X,Y )
)
µ(X,Y
=
, Σm
=
(4)
m
)
(Y X)
(Y Y )µ(YΣmΣmm
where N (·; µ, Σ) denotes a Gaussian distribution with a mean
vector µ and a covariance matrix Σ. The mixture component
index is m. The total number of mixture components is M .
A parameter set of the GMM is λ, which consists of mixture)component weights αm , mean vectors µ(X,Yand full covariancem(X,Y )
matrices Σmfor individual mixture components. The mean)
vector µ(X,Yconsists of an input mean vector µ(X)mm and an out(X,Y ))put mean vector µ(Y.
The
covariancematrixΣ
consistsmm(XX)(Y Y )
of input and output covariance matrices Σmand Σm
and
(XY )
(Y X)
cross-covariance matrices Σmand Σm . We also train a
Gaussian distribution modeling the probability density of the GV
for the spectrum parameter of the target normal speech.
3.2. Conversion Process
Individual speech parameters of the target normal speech are independently estimated from the spectral segment features extracted
from the EL speech using each of the trained GMMs as follows:ŷ = argmax P (Y |X, λ)P (v(y)|λ(v) )ω
ysubject to Y = W y

(5)

⊤
⊤ ⊤
⊤
⊤
where X = [X ⊤1 , · · · , X t , · · · , X T ] , Y = [Y 1 , · · · , Y t ,
⊤
⊤
⊤ ⊤
⊤ ⊤
· · · , Y T ] , and ŷ = [ŷ 1 , · · · , ŷ t , · · · , ŷ T ] are time sequence
vectors of the input spectral segment features, the output features,and the converted static features of each target speech parameterover an utterance, respectively. The matrix W is a transform toextend the static feature vector sequence into the joint static anddynamic feature vector sequence [12]. The GV probability density is given by P (v(y)|λ(v) ), where v(y) is the GV of the target static feature vector sequence y and λ(v) is a parameter set of
the Gaussian distribution for the GV. The GV likelihood weightis given by ω. The GV likelihood is usually considered only inthe spectral estimation, i.e., ω is set to zero in the F0 estimationand the aperiodic estimation. After estimating time sequences of
the converted spectrum, F0 , and aperiodic components, a mixed
excitation signal is generated using the converted F0 and aperiodic components [13]. Finally, the converted speech signal issynthesized by filtering the generated excitation signal with theconverted spectral parameters.

Figure 2: EL speech enhancement based on SS.

3. Electrolaryngeal Speech Enhancement
Based on Statistical Voice Conversion (VC)
EL speech enhancement based on VC [2] attempts to convert EL
speech of laryngectomees into normal speech of non-disabled
speakers. It consists of training and conversion processes, as
shown in Figure 3. To achieve the conversion from EL speech
into normal speech, three conversion models are used to separately estimate spectrum, F0 , and aperiodic components, which
capture the noise strength of an excitation signal on each frequency band [9]. These models are trained in advance using aparallel data set consisting of utterance pairs of a laryngectomee
and a target non-disabled speaker. Conversion employs maximum

3068

Figure 3: EL speech enhancement based on VC.

4. Hybrid Approach to Electrolaryngeal
Speech Enhancement
The SS-based EL speech enhancement method essentially estimates EL speech produced by the lips while reducing the impactof leaked excitation sounds. Even if the leaked excitation sounds
are completely removed, improvements in naturalness yielded bythis method will be small because the produced EL speech intrinsically suffers from the lack of naturalness caused by highly artificial F0 patterns and the mechanical excitation sound quality. On
the other hand, this method does not cause any significant degradation in intelligibility of EL speech. In other words, this method
may cause small improvements, but very rarely degradations inspeech quality.
The VC-based EL speech enhancement method has the potential to significantly improving naturalness of EL speech by converting EL speech into normal speech. As the converted speech
signal is generated from statistics of normal speech parameters,
it does not suffer from the artificial F0 patterns and mechanicalsound quality. However, the conversion process in this method isquite complex, and therefore, errors in conversion are inevitable.
These errors tend to cause degradation in intelligibility of converted speech as adverse effects.
In order to develop an EL speech enhancement method thatallows for the large improvements of naturalness realizable by VC
while ameliorating its adverse effects, we propose a hybrid approach based on SS and VC. The proposed EL speech enhancement method is shown in Figure 4. As laryngectomees have thecapability to properly articulate the excitation signals, spectral parameters of EL speech do not need to be changed greatly to generate intelligible speech. Therefore, we use the spectral parametersrefined with SS without applying VC. On the other hand, it isessentially difficult to generate natural excitation signals exhibit-

Figure 4: EL speech enhancement based on the proposed hybrid
approach.
ing natural F0 patterns in EL speech production. Therefore, we
use VC to estimate the excitation parameters: i.e., F0 and aperiodic components. The proposed hybrid method can be expected
to yield much larger improvements in naturalness compared with
the SS-based enhancement method thanks to the use of more natural excitation signals generated from statistics of normal speech.
It also can be expected to alleviate the degradation in intelligibility observed in the conventional VC-based enhancement methodby avoiding errors in spectral conversion.
In the excitation parameter estimation based on VC, unvoiced/voiced (U/V) information is also predicted in the manner described in [10]. However, as EL speech is totally voiced
speech, it is possible that significant improvements in naturalness can be yielded even if U/V information is not added to theconverted speech. To further reduce the possibility of degradation in intelligibility caused by the U/V prediction errors, we also
propose the use of continuous F0 patterns without any unvoiced
frames to generate the excitation signals. In the training process,continuous F0 patterns of normal speech are generated by using
spline interpolation to add F0 values to unvoiced frames, and the
GMM is trained on this modified data. In the conversion process, continuous F0 patterns are estimated over all frames. As
it is straightforward to automatically detect silence frames in EL
speech simply using waveform power, unvoiced excitation signals are generated only at those frames. At the other voice active

3069

frames, voiced excitation signals are always generated. Althoughunvoiced phoneme sounds cannot be generated in this method, theconverted speech does not suffer from wrongly predicting voiced
frames as unvoiced frames. Because unvoiced phoneme sounds
also can not be generated in the original EL speech, this method
causes no adverse effect.

Table 1: Conversion accuracy in enhancement methods with VC.
VC
Mel-cepstral distortion
without power infomation
U to V error rate
V to U error rate
F0 correlation coefficient
Aperiodic distortion

5. Experimental Evaluation
5.1. Experimental Conditions
In our experiments, the source speaker was a laryngectomee andthe target speaker was a non-disabled speaker. Both speakers
recorded 50 phoneme-balanced sentences. We conducted a 5fold cross validation test in which 40 utterance pairs were used
for training, and the remaining 10 utterance-pairs were used forevaluation. Sampling frequency was set to 16 kHz. In the VCbased enhancement methods, the 0th through 24th mel-cepstralcoefficients extracted by STRAIGHT analysis [14] were used as
the spectral parameters. The shift length was set to 5 ms. For thesegment feature extraction, current ± 4 frames were used. The
numbers of mixture components were set to 32 for the spectral
and aperiodic estimation, 64 for the F0 estimation, and 32 forcontinuous F0 estimation.
We conducted both objective and subjective evaluations. In
the objective evaluation, conversion accuracy in the VC-based enhancement method was evaluated using mel-cepstral distortion,
the U/V error rate, F0 correlation coefficient, and aperiodic distortion between the converted speech parameters and the natural
target speech parameters. In the subjective evaluations, we conducted two opinion tests of intelligibility and naturalness using a
5-scaled opinion score (1: Bad, 2: Poor, 3: Fair, 4: Good, and
5: Excellent). The following five types of speech samples were
evaluated by 5 listeners:
EL original EL speech

SS+VC

SS+VC+CF0

5.09 dB———————
4.04 %
17.9 %
4.42 %
0.09 %
0.53
0.55
3.19 dB

Figure 5: Result of opinion test on intelligibility.

Figure 6: Result of opinion test on naturalness.

SS speech enhanced by the SS-based enhancement method

Figure 6 shows a result of the opinion test on naturalness.
SS yields a very small improvement in naturalness. On the other
hand, VC yields a significantly larger improvement. The proposed hybrid methods (SS+VC and SS+VC+CF0) also yield significantly larger improvements compared with SS as they are capable of generating more natural F0 patterns. We can also observethat the continous F0 estimation is effective for improving naturalness as well.
These results suggest that the proposed hybrid approach to
EL speech enhancement based on the continous F0 estimation
is effective in significantly improving naturalness of EL speech
while avoiding degradation in intelligibility that is often observedin the conventional VC-based enhancement method.

VC speech enhanced by the VC-based enhancement method
SS+VC speech enhanced by the proposed hybrid enhancement
method with U/V prediction
SS+VC+CF0 speech enhanced by the proposed hybrid enhancement method with continuous F0 estimation
5.2. Experimental Results
First, we show results of the objective evaluation for conversion
accuracy for the enhancement methods with VC in Table 1. It
can be observed that the F0 correlation coefficient is improved
slightly by the continous F0 estimation. We have found that large
errors in the F0 estimation tend to be observed at short voiced
segments that are sometimes generated in VC or SS+VC. This improvement is similar to that yielded by the continuous F0 modeling in HMM-based speech synthesis [15]. Moreover, it is reasonable that the U to V error rate increases and the V to U error rate
decreases in SS+VC+CF0 compared with those in VC or SS+VC.
The V to U errors still exist in SS+VC+CF0 owing to errors in theautomatic silence frame detection with waveform power but they
are almost negligble.
Next, in Figure 5 we show the results of the subjective opinion test on intelligibility. It can be seen that a slight improvement
is yielded by SS. On the other hand, VC causes significant degradation as reported in [3]. SS+VC doesn’t cause degradation compared with EL but it still causes very small degradation comparedwith SS. This adverse effect on intelligibility is not observed in
the proposed hybrid methods (SS+VC and SS+VC+CF0) thanks
to no spectral conversion error.

6. Conclusions
In this paper, we have proposed a hybrid approach to electrolaryngeal (EL) speech enhancement based on spectral subtraction
for spectral parameter estimation and statistical voice conversion
for excitation parameter prediction. To further avoid conversion
errors causing degradation in intelligibility, the continuous F0 estimation method has also been implemented for the proposed approach. As a result of an experimental evaluation, it has been
demonstrated that the proposed approach is capable of significantly improving naturalness of EL speech while causing no adverse effect such as the degradation in intelligibility.

7. Acknowledgements
Part of this work was supported by JSPS KAKENHI Grant Number 22680016.

3070

8. References[1] H. Liu, Q. Zhao, M.X. Wan, and S.P. Wang, “Enhancement of electrolarynx speech based on auditory masking,” IEEE Trans. Biomedical Engineering, vol. 53, no. 5, pp. 865–874, May 2006.
[2] K. Nakamura, T. Toda, H. Saruwatari, K. Shikano, “Speaking-aid
systems using GMM-based voice conversion for electrolaryngeal
speech,” SPECOM, vol. 54, no. 1, pp. 134–146, Jan 2012.
[3] H. Doi, K. Nakamura, T. Toda, H. Saruwatari, and K. Shikano,“An evaluation of alaryngeal speech enhancement methods based onvoice conversion techniques,” Proc. ICASSP, pp. 5136–5139, May
2011.
[4] S.F. Boll, “Suppression of acoustic noise in speech using spectral
subtraction,” IEEE Trans. Acoustics, Speech and Signal Processing,vol. 27, no. 2, pp. 113–120, Apr 1979.
[5] Y. Stylianou, O. Cappe, and E. Moulines, “Continuous probabilistic transform for voice conversion,” IEEE Trans. Speech and Audio
Processing, vol. 6, no. 2, pp. 131–142, Mar 1998.
[6] T. Toda, A.W. Black, and K. Tokuda, “Voice conversion based on
maximum likelihood estimation of spectral parameter trajectory,”
IEEE Trans. Audio, Speech, and Language, Vol. 15, No. 8, pp.
2222–2235, Nov 2007.
[7] H.R. Sharifzadeh, I.V. McLoughlin, and F. Ahmadi, “Reconstruction of normal sounding speech for laryngectomy patients through a
modified CELP codec,” IEEE Trans. Biomedical Engineering, vol.
57, no. 10, pp. 2448–2458, Oct 2010.
[8] B.L. Sim, Y.C. Tong, J.S. Chang, and C.T. Tan, “A parametricformulation of the generalized spectral subtraction method,” IEEE
Trans. Speech and Audio Processing, Vol. 6, No. 4, pp. 328–337, Jul
1998.
[9] H. Kawahara, J. Estill, and O. Fujimura, “Aperiodicity extraction
and control using mixed mode excitation and group delay manipulation for a high quality speech analysis, modification and system
STRAIGHT,” Proc. 2nd MAVEBA, Sep 2001.
[10] T. Toda, M. Nakagiri, and K. Shikano, “Statistical voice conversion techniques for body-conducted unvoiced speech enhancement,”
IEEE Trans. Audio, Speech, and Language, Vol. 20, No. 9, pp.
2505–2517, Nov 2012.
[11] A. Kain and M. W. Macon, “Spectral voice conversion for text-tospeech synthesis,” Proc. ICASSP, pp. 285–288, May 1998.
[12] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura, “Speech parameter generation algorithms for HMM-based
speech synthesis,” Proc. ICASSP, pp. 1315–1318, June 2000.
[13] Y. Ohtani, T. Toda, H. Saruwatari, K. Shikano, “Maximum likelihood voice conversion based on GMM with STRAIGHT mixed
excitation,” Proc. Interspeech, pp. 2266–2269, Sep 2006.
[14] H. Kawahara, I. Masuda-Katsuse, and A. Cheveigne, “Restructuring speech representations using a pitch-adaptive time-frequency
smoothing and an instantaneous-frequency-based F0 extraction:
Possible role of a repetitive structure in sounds,” SPECOM, Vol. 27,
No. 3-4, pp. 187–207, Apr 1999.
[15] K. Yu and S. Young, “Continuous F0 modelling for HMM based
statistical parametric speech synthesis,” IEEE Trans. Audio, Speech,
and Language, Vol. 19, No. 5, pp. 1071–1079, Jul 2011.

3071