IEICE TRANS. INF. & SYST., VOL.E97–D, NO.6 JUNE 2014

1429

PAPER

Special Section on Advances in Modeling for Real-world Speech Information Processing and its Application

A Hybrid Approach to Electrolaryngeal Speech Enhancement
Based on Noise Reduction and Statistical Excitation Generation
Kou TANAKA†a) , Nonmember, Tomoki TODA†b) , Member, Graham NEUBIG†c) , Nonmember,
Sakriani SAKTI†d) , and Satoshi NAKAMURA†e) , Members

SUMMARY
This paper presents an electrolaryngeal (EL) speech enhancement method capable of significantly improving naturalness of EL
speech while causing no degradation in its intelligibility. An electrolarynx is an external device that artificially generates excitation sounds to enable laryngectomees to produce EL speech. Although proficient laryngectomees can produce quite intelligible EL speech, it sounds very unnaturaldue to the mechanical excitation produced by the device. Moreover, theexcitation sounds produced by the device often leak outside, adding to EL
speech as noise. To address these issues, there are mainly two conventional
approached to EL speech enhancement through either noise reduction or
statistical voice conversion (VC). The former approach usually causes nodegradation in intelligibility but yields only small improvements in naturalness as the mechanical excitation sounds remain essentially unchanged. On
the other hand, the latter approach significantly improves naturalness of EL
speech using spectral and excitation parameters of natural voices converted
from acoustic parameters of EL speech, but it usually causes degradation
in intelligibility owing to errors in conversion. We propose a hybrid approach using a noise reduction method for enhancing spectral parametersand statistical voice conversion method for predicting excitation parameters. Moreover, we further modify the prediction process of the excitation parameters to improve its prediction accuracy and reduce adverse effects caused by unvoiced/voiced prediction errors. The experimental results
demonstrate the proposed method yields significant improvements in naturalness compared with EL speech while keeping intelligibility high enough.
key words: speaking-aid, electrolaryngeal speech, spectral subtraction,
voice conversion, hybrid approach

1.

Introduction

Speech is one of the most common media of human communication. Unfortunately, there are many people with disabilities that prevent them from producing speech freely,leading to communication barriers. One example of people
who cannot produce speech freely is laryngectomees, whohave undergone an operation to remove the larynx including the vocal folds for reasons such as an accident or laryngeal cancer. Laryngectomees cannot produce speech in theusual manner because they no longer have their vocal folds.
Therefore, they require another method to produce speechwithout the vocal fold vibration.
Electrolaryngeal (EL) speech is produced by one of theManuscript received September 28, 2013.
Manuscript revised January 18, 2014.
†
The authors are with Nara Institute of Science and Technology (NAIST), Ikoma-shi, 630–0192 Japan.
a) E-mail: ko-t@is.naist.jpb) E-mail: tomoki@is.naist.jpc) E-mail: neubig@is.naist.jpd) E-mail: ssakti@is.naist.jpe) E-mail: s-nakamura@is.naist.jp
DOI: 10.1587/transinf.E97.D.1429

Fig. 1 Speech production mechanisms of non-disabled people (left figure) and total laryngectomees (right figure).

major alternative speaking methods for laryngectomees asshown in Fig. 1. EL speech is produced using an electrolarynx, which is an electromechanical vibrator that is typically held against the neck to mechanically generate artificial excitation signals. The generated excitation signals are
conducted into the speaker’s oral cavity, and EL speech is
produced by articulating the conducted excitation signals.
There are several advantages of EL speech compared withother types of alaryngeal speech, such as esophageal speech:
e.g., 1) it is easy to learn how to produce EL speech, 2)less physical power is needed to produce EL speech, and
3) EL speech is relatively intelligible. However, there arealso some issues of EL speech: e.g., 1) the excitation sounds
are usually emitted outside as noise causing degradation ofsound quality, and 2) naturalness is very low owing to its
mechanical sound quality caused by the mechanically generated excitation signals. In particular, the latter issue is an
essential drawback of EL speech caused by the diﬃculty ofartificially generating natural F0 patterns corresponding tolinguistic content.
To address these issues of EL speech, two approaches
have conventionally been adopted. One is based on noise
reduction [1] and the other is based on statistical voice conversion (VC) [2], [3]. The former approach aims to reduce
the eﬀect of the excitation sounds leaked from the electrolarynx by using noise reduction techniques, such as spectral
subtraction (SS) [4]. This noise reduction process causes nodegradation in intelligibility but yields only small improvements in naturalness as the mechanical excitation sounds
remain essentially unchanged. On the other hand, the latter method is capable of significantly improving natural-c 2014 The Institute of Electronics, Information and Communication Engineers
Copyright IEICE TRANS. INF. & SYST., VOL.E97–D, NO.6 JUNE 2014

1430

ness by converting acoustic parameters of EL speech into
those of natural voices using statistical VC techniques [5],[6]. The use of statistics extracted from a parallel data set
consisting of EL speech and natural voices makes it possible to achieve more complex conversion processes than thatof other signal processing approaches, such as formant manipulation [7]. For example, it is possible to convert from aspectral parameter sequence of EL speech into F0 patternsof natural voices. However, VC-based approaches usuallycause degradation in intelligibility owing to errors in conversion [3]. Even if naturalness is significantly improved
compared to EL speech, the enhanced speech suﬀering fromdegradation from another perspective will have less chanceof being accepted by actual users. In particular, intelligible
speech production is the most essential factor for laryngectomees to communicate with others. Therefore, it is required
to develop an EL speech enhancement technique causing
no degradation in intelligibility compared to the original EL
speech.
In this paper, to develop an EL speech enhancement
method for significantly improving naturalness while preserving intelligibility in EL speech, we propose a hybrid
method using the SS-based noise reduction method for enhancing spectral parameters and the VC method for predicting excitation parameters. Furthermore, to improve prediction accuracy of the excitation parameters and reduce adverse eﬀects caused by unvoiced/voiced prediction errors,
we modify the prediction process of the excitation parameters. We conduct an experimental evaluation, which demonstrates that the proposed method yields significant improvements in naturalness compared with EL speech while causing no degradation in intelligibility.
2.

Conventional EL Speech Enhancement

2.1

Enhancement Based on Spectral Subtraction (SS)EL speech enhancement based on SS.

domain parameter, and L̂(ω) is an estimate of the averaged
amplitude spectrum of the additive noise signal. The enhanced speech signal is generated using the processed amplitude spectrum and the original phase extracted from thenoisy speech signal.
In this paper, we implement SS for EL speech enhancement, as shown in Fig. 2. The averaged amplitude spectrumof the additive noise signal is estimated in advance using theexcitation signals generated from the electrolarynx. In orderto record only the excitation signals leaked from the electrolarynx as accurately as possible, the excitation signals
are recorded with a close-talking microphone while keeping speaker’s mouth closed. The excitation signals are generated with the electrolarynx held in the usual manner, as
shown in Fig. 1.
2.2 Enhancement Based on Statistical Voice Conversion(VC)

SS is a method for restoration of the amplitude spectrumof a speech signal that has been observed together with theleaked noise of an electrolarynx. This is done by subtracting
an estimate of the amplitude spectrum of the noise from theamplitude spectrum of the noisy speech signal. The noisy
speech signal model in the frequency domain is expressed
as follows:Y(ω, t) = S (ω, t) + L(ω, t)

Fig. 2(1)

where Y(ω, t), S (ω, t), and L(ω, t) are respectively components of the noisy speech signal, the clean speech signal,and the additive noise signal at frequency ω and time framet. Assuming that the additive noise signal is stationary, thegeneralized SS scheme [8] is described as follows:⎧|L̂(ω)|γ1⎪γ
γ⎪( |Y(ω,t)|γ < α+β )γ ⎨|Y(ω, t)| − α| L̂(ω)|(2)|Ŝ (ω, t)| = ⎪⎪⎩β|L̂(ω)|γ(otherwise)
where α (α > 1) is an over-subtraction parameter, β (0 ≤β ≤ 1) is a spectral flooring parameter, γ is an exponential

EL speech enhancement based on VC [2] attempts to convert EL speech of laryngectomees into normal speech ofnon-disabled speakers. It consists of training and conversion processes, as shown in Fig. 3. To achieve the conversion from EL speech into normal speech, three conversion models are used to separately estimate spectrum, F0 ,and aperiodic components, which capture the noise strengthof an excitation signal on each frequency band [9]. Thesemodels are trained in advance using a parallel data set consisting of utterance pairs of a laryngectomee and a target
non-disabled speaker. Conversion employs maximum likelihood estimation of speech parameter trajectories considering global variance (GV) [6]. This framework is the sameas in conversion from body-conducted unvoiced speech into
normal speech [10].
2.2.1

Training Process

The spectral components of EL speech are unstably andspectral structures of some phonemes are often collapsed
due to the production mechanism of EL speech. To address these issues, we use spectral segment features ex-

TANAKA et al.: A HYBRID APPROACH TO ELECTROLARYNGEAL SPEECH ENHANCEMENT

1431

bility densities [11] of the spectral segment features of EL
speech and each of the output feature vectors of individual
target parameters of normal speech using the corresponding
joint feature vector set as follows:
P(Xt , Y t |λ)
M



=
αm N [Xt , Y t ] ; μ(X,Y), Σ(X,Y)
m
m

(4)

m=1

=
μ(X,Y)
m

μ(X)
m
(Y)
μm

=
, Σ(X,Y)
m

Σ(XX)
m
(Y X)
Σm

Σ(XY)
m
Σ(YY)
m

(5)

where N(·; μ, Σ) denotes a Gaussian distribution with amean vector μ and a covariance matrix Σ. The mixture component index is m. The total number of mixture components
is M. A parameter set of the GMM is λ, which consists
andof mixture-component weights αm , mean vectors μ(X,Y)
m
for individual mixture comfull covariance matrices Σ(X,Y)
mconsists of an input meanponents. The mean vector μ(X,Y)m(Y)
vector μ(X)m and an output mean vector μm . The covariance(X,Y)
matrix Σm consists of input and output covariance matriand Σ(YY)
and cross-covariance matrices Σ(XY)andces Σ(XX)
m
m
m
(Y X)
Σm . We also train a Gaussian distribution modeling theprobability density of the GV for the spectrum parameter of
the target normal speech.
2.2.2

Conversion Process

Individual speech parameters of the target normal speech
are independently estimated from the spectral segment features extracted from the EL speech using each of the trained
GMMs as follows:ŷ = argmax P(Y|X, λ)P(u(y)|λ(v) )ω
ysubject to Y = Wy
Fig. 3EL speech enhancement based on VC.

tracted from multiple frames as follows:
Xt = CX t + d(3)

where X t = [xt−i , · · · , xt , · · · , xt+i ] is a joint vector generated by concatenating a spectral parameter vector xt at
the current frame and those at ±i preceding and succeeding frames. Because this joint vector includes redundant information, dimension reduction with principal component
analysis (PCA) is performed for the joint vector X t in order to extract spectral segment features Xt at frame t, where
C and d are a transformation matrix and a bias vector extracted by PCA, respectively. On the other hand, we assume
a static feature vector yt of each type of the normal speechparameters at frame t. As an output speech feature vector,
we use Y t = [yt , Δyt ] consisting of the static and dynamic
features, where  denotes transposition of the vector. We
independently train three GMMs to model the joint proba-(6)

where X = [X1 , · · · , Xt , · · · , XT ] , Y = [Y 1 , · · · , Y t , · · · ,Y T ] , and ŷ = [ŷ1 , · · · , ŷt , · · · , ŷT ] are time sequence
vectors of the input spectral segment features, the output
features, and the converted static features of each target
speech parameter over an utterance, respectively. The matrix W is a transform to extend the static feature vector sequence into the joint static and dynamic feature vector sequence [12]. The GV probability density function is givenby P(u(y)|λ(v) ), where u(y) is the GV of the target static feature vector sequence y and λ(v) is a parameter set of theGaussian distribution for the GV. The GV likelihood weightis given by ω, which is set to the ratio of the number ofdimensions between vector u(y) and Y, i.e., 1/(2T ) in thispaper. The GV likelihood is usually considered only in the
spectral estimation, i.e., ω is set to zero in the F0 estimation and the aperiodic estimation. After estimating time sequences of the converted spectrum, F0 , and aperiodic components, a mixed excitation signal is generated using theconverted F0 and aperiodic components [13]. Finally, theIEICE TRANS. INF. & SYST., VOL.E97–D, NO.6 JUNE 2014

1432

converted speech signal is synthesized by filtering the generated excitation signal with the converted spectral parameters.
3.

Proposed EL Speech Enhancement Based on a Hybrid Approach

3.1

Enhancement based on a Hybrid Approach

The SS-based EL speech enhancement method essentiallyestimates EL speech produced by the lips while reducing
the impact of leaked excitation sounds. Even if the leaked
excitation sounds are completely removed, improvements innaturalness yielded by this method will be small because theproduced EL speech intrinsically suﬀers from the lack ofnaturalness caused by highly artificial F0 patterns and themechanical excitation sound quality. On the other hand,
this method does not cause any significant degradation in
intelligibility of EL speech. In other words, this method
may cause small improvements, but very rarely degradations in speech quality. The VC-based EL speech enhancement method has the potential to significantly improve naturalness of EL speech by converting EL speech into normal
speech. As the converted speech signal is generated fromstatistics of normal speech parameters, it does not suﬀer
from the artificial F0 patterns and mechanical sound quality. However, the conversion process in this method is quite
complex, and therefore, errors in conversion are inevitable.
These errors tend to cause degradation in intelligibility of
converted speech as adverse eﬀects.
In order to develop an EL speech enhancement methodthat allows for the large improvements of naturalness realizable by VC while ameliorating its adverse eﬀects, we propose a hybrid approach based on SS and VC. The proposed
EL speech enhancement method is shown in Fig. 4. As laryngectomees have the capability to properly articulate theexcitation signals, spectral parameters of EL speech do nothave to be changed greatly to generate intelligible speech.
Therefore, we use the spectral parameters refined with SS
without applying VC. On the other hand, it is essentially
diﬃcult to generate excitation signals exhibiting natural F0
patterns in EL speech production. Therefore, we use VC
to estimate the excitation parameters: i.e., F0 and aperiodic
components. The proposed hybrid method can be expected
to yield much larger improvements in naturalness comparedwith the SS-based enhancement method thanks to the useof more natural excitation signals generated from statistics
of normal speech. It also can be expected to alleviate thedegradation in intelligibility observed in the conventional
VC-based enhancement method by avoiding errors in spectral conversion.
3.2

Improvement of Statistical Excitation Prediction

We propose several methods to improve statistical excitation
prediction in the hybrid enhancement process.

Fig. 4EL speech enhancement based on the proposed hybrid approach.

3.2.1

Continuous F0 Patterns (CF0)The F0 patterns extracted from natural voices are discontinuous because the F0 values are not observed at the unvoiced and silent frames. In the conventional VC-based enhancement method, a constant value clearly diﬀerent fromF0 values (e.g., a value much less than the minimum F0
value) is used to represent F0 values at those frames, andboth unvoiced/voiced prediction and F0 value estimation are
performed with a single GMM in the same manner as described in [10]. However, it is not straightforward to accurately model such a discontinuous F0 pattern.
To simplify characteristics of the parameter sequence
to be modeled, we apply a continuous F0 pattern to the statistical excitation prediction. In the training process, con-

TANAKA et al.: A HYBRID APPROACH TO ELECTROLARYNGEAL SPEECH ENHANCEMENT

1433

tinuous F0 patterns of normal speech are generated by using spline interpolation to produce F0 values at unvoiced
frames, as shown in the middle of Fig. 5. The resulting continuous F0 patterns are used as the target parameter in the
GMM training. The conventional discontinuous F0 patterns
are also modeled with another GMM to predict U/V information. In the conversion process, a continuous F0 pattern
and U/V information are separately predicted using the corresponding GMMs. A discontinuous F0 pattern to be usedin synthesis is finally generated by combining them. The effectiveness of using the continuous F0 pattern has also beenreported in the field of statistical parametric speech synthesis [14].
3.2.2

Remove Micro-Prosody with Low-Pass Filter (LPF)

Rapid movements, called micro-prosody, are often observedin F0 patterns extracted from natural voices. However, it
is diﬃcult to accurately model and reproduce these movements with a GMM. Moreover, an impact of micro-prosody
on naturalness of synthetic speech is much smaller than thatof F0 patterns corresponding to phrase and accentual components. Therefore, it is helpful to make the GMM focus onmodeling only those patterns. To achieve this, we propose
the use of a method to smooth the continuous F0 patterns
with low-pass filtering [15] as shown in the middle of Fig. 5.
The smoothed continuous F0 patterns are then modeled with
the GMM.
3.2.3

Avoiding U/V Prediction Errors

In the excitation parameter prediction, U/V information is
also predicted as mentioned above. Errors during this pre-Fig. 5 Each type of F0 patterns. Top figure is target F0 counters, middle is continuous F0 patterns using the spline interpolation, and bottom issmoothing continuous F0 patterns using the low-pass filter.

diction process are also unavoidable and they may cause adverse eﬀects in intelligibility.
As EL speech is totally voiced speech, no degradation
is caused even if the converted speech is generated by regarding all speech frames as voiced frames. To further reduce the possibility of degradation in intelligibility causedby U/V prediction errors, we also propose the use of continuous F0 patterns without any unvoiced frames for speechsegments to generate the excitation signals. In the conversion process, continuous F0 patterns are predicted over all
frames. Then, only silence frames are automatically detected using waveform power and unvoiced excitation signals are generated only at those frames. Unvoiced phoneme
sounds cannot be generated in this method as in the original EL speech but the converted speech does not suﬀer fromwrongly predicted unvoiced frames.
4.

Experimental Evaluations

4.1 Experimental Conditions
We objectively and subjectively compared the performanceof the proposed system with the conventional systems. In
our experiments, the source speaker was one laryngectomee
and the target speaker was one non-disabled speaker. Bothspeakers recorded 50 phoneme-balanced sentences. We
conducted a 5-fold cross validation test in which 40 utterance pairs were used for training, and the remaining 10
utterance-pairs were used for evaluation. Sampling frequency was set to 16 kHz. In the VC-based enhancement methods, the 0th through 24th mel-cepstral coeﬃcients extracted by STRAIGHT analysis [16] were used as
the spectral parameters. The shift length was set to 5 ms.
STRAIGHT analysis was also used for spectral extractionof EL speech but F0 values were set to 100 Hz without F0
extraction, which was equivalent to F0 values of excitation
signals generated by the electrolarynx used in the experiments. For the segment feature extraction, the current ± 4
frames were used to extract a 50-dimensional feature vector.
PCA was conducted to determine the transformation matrix
and the bias vector in Eq. (3) using all EL speech samples
in the training data. The numbers of mixture components
were set to 32 for the spectral and aperiodic estimation, 64
for the F0 estimation, and 32 for continuous F0 estimation.
In the SS-based spectral enhancement method, the numberof FFT points was set to 512 and individual parameters were
set to α = 2.0, β = 0, 0, and γ = 1.0, which were manually
determined by listening to the enhanced speech so that its
voice quality was improved as much as possible. The cutoﬀ frequency of the low-pass filter was set to 10 Hz. Notethat in the VC-based enhancement method, mel-cepstral andaperiodic distortion is shown in Table 1.

Table 1

Conversion accuracy in enhancement methods with VC.

Mel-cepstral distortion without power information
Aperiodic distortion

5.09 dB
3.19 dB

IEICE TRANS. INF. & SYST., VOL.E97–D, NO.6 JUNE 2014

1434

Fig. 6

4.2

Conversion accuracy for the F0 patterns of each systems.

Objective Evaluations

We evaluated the eﬀectiveness of the proposed preprocessing for the training data, including the continuous F0 estimation method and the low-pass filter. Note that the eﬀectiveness of the continuous F0 estimation method has alreadybeen reported in [14]. As measures to evaluate the prediction accuracy of the excitation features, we used the correlation coeﬃcient and U/V error rate on F0 components between the converted speech parameters and the natural target
speech parameters. As for the evaluation of the F0 correlation coeﬃcient, we set the number of GMM mixture components to 8, 16, 32, or 64. We evaluate three systems using
the normal F0 patterns extracted from target natural voices,
the continuous F0 patterns interpolated with the normal F0
patterns using spline interpolation, and the smoothing continuous F0 patterns extracted from continuous F0 patterns
through the low-pass filter. On the other hand, for the evaluation of U/V error rate, we also set the number of GMM
mixture components for VC to 8, 16, 32, or 64.
Figure 6 shows the result of the evaluation for the F0
correlation coeﬃcient. In the case of using only the VC
method, the F0 correlation coeﬃcient depends on the number of GMM mixture components, and is maximized with 64
mixture components. However, as for the proposed method,
VC + CF0 and VC + CF0 + LPF, those are not depended
well. Especially, in the small number of GMM mixture
components, a significant degradation is not observed compared with using only VC methods. Moreover, it can beobserved that the F0 correlation coeﬃcient is improved by
the continuous F0 estimation and also improved by using the
low-pass filter. Note that in the case of the use of the continuous F0 estimation method and the low-pass filter, the F0
correlation coeﬃcient is maximized with 32 mixture components.
Figure 7 shows the result of the evaluation for U/V error rate. We have found that large errors in the F0 estimation
tend to be observed at short voiced segments that are sometimes generated in only the VC-based enhancement method.
This improvement is similar to that yielded by the continuous F0 modeling in HMM-based speech synthesis [14]. As

Fig. 7

U/V error rate of each systems.

the number grows larger, V-to-U error rate decreases while
U-to-V increases. With 64 mixture components, the U/Verror rate is minimized. On the other hand, without U/Vprediction, the U/V error rate is constant. In particular, theV-to-U error rate is practically zero. The V-to-U errors still
exist without the continuous F0 estimation method owing toerrors in the automatic silence frame detection with waveform power, but they are almost negligible. However, U-toV significantly increases owing to the continuous F0 estimation method. Note that as we mentioned in Sect. 3.2.3, thisincrease causes no adverse eﬀect compared with EL speech
because EL speech is totally voiced speech.
4.3 Subjective Evaluations
We conducted two opinion tests of listenability and naturalness and a dictation test on intelligibility. In this paper,
the term “listenability” is used to indicate a score that wasmeasured by asking the listener to subjectively evaluate howeasy it was to understand the utterance. The term “intelligibility” is used to indicate a score that was calculated by asking the listener to write down the content of the utterance,and measuring the accuracy of transcription. The term “naturalness” is used to indicate a score that was measured byasking the listener to subjectively evaluate whether the evaluated speech is similar to natural human speech or not. In
the opinion tests, each listener evaluated each speech quality of the enhanced voices using a 5-scaled opinion score(1: Bad, 2: Poor, 3: Fair, 4: Good, and 5: Excellent). We
evaluated the following five types of speech samples:EL original EL speech
SS speech enhanced by the SS-based enhancement method
VC speech enhanced by the VC-based enhancement
methodSS+VC speech enhanced by the proposed hybrid enhancement method with U/V prediction
SS+VC+CF0 speech enhanced by the proposed hybrid enhancement method with continuous F0 estimation
On the other hand, in the dictation test, in order todemonstrate the eﬀectiveness of avoiding U/V prediction errors, we evaluated the following five types of speech sam-

TANAKA et al.: A HYBRID APPROACH TO ELECTROLARYNGEAL SPEECH ENHANCEMENT

1435

Fig. 8Result of opinion test on listenability.

Fig. 9Result of opinion test on naturalness.

Fig. 10

Result of opinion test on naturalness.

ples:
EL original EL speech
SS speech enhanced by the SS-based enhancement method
Hybrid(V) speech enhanced by the proposed hybrid enhancement method with continuous F0 estimation
Hybrid (U/V) speech enhanced by the proposed hybrid enhancement method with U/V prediction
Hybrid (target U/V) speech enhanced by the proposed hybrid enhancement method with ideal U/V information
where the proposed hybrid enhancement method is a method
based on SS+VC+CF0+LPF. As the reference U/V information, we use target U/V information obtained by performing DTW between the enhanced speech parameters using
the VC-based enhancement method and the natural target
speech parameters. Intelligibility was evaluated using wordcorrect rate and word accuracy, which were calculated asfollows:C
S +D+CC−I
word accuracy [%] =S +D+C
word correct rate [%] =

(7)
(8)

where S is the number of substitutions, D is the number ofdeletions, I is the number of insertions, and C is the number of correct words. Note that the VC-based enhancement
method generally causes a significant degradation in intelligibility (around 3% word recognition rate reduction) compared to EL speech as reported in [3]. All tests were performed by 5 listeners. Each listener evaluated 50 samples,
10 samples per system † .
First, in Fig. 8, we show the results of the subjective
opinion test on listenability. It can be seen that a slight improvement is yielded by SS. On the other hand, VC causes
significant degradation as reported in [3]. SS+VC doesn’t
cause a degradation compared with EL but it still causes avery small degradation compared with SS. This adverse effect on listenability is not observed in the proposed hybrid
methods (SS+VC and SS+VC+CF0) thanks to no spectral
conversion error.
†
Some samples are available at the URL: http://isw3.naist.jp/
˜ko-t/eval/publication/IEICE/HAESE/index.html

Figure 9 shows a result of the opinion test on naturalness. SS yields a very small improvement in naturalness. On the other hand, VC yields a significantly larger
improvement. The proposed hybrid methods (SS+VC and
SS+VC+CF0) also yield significantly larger improvements
compared with SS as they are capable of generating morenatural F0 patterns. We can also observe that the continuous
F0 estimation is eﬀective for improving naturalness as well.
Figure 10 shows a result of the dictation test on intelligibility. We found that the hybrid methods do not cause any
degradation in intelligibility compared with EL speech. Furthermore, in the hybrid method that avoided U/V prediction
by using the continuous F0 estimation method, the intelligibility is preserved, similarly to the hybrid method using
ideal U/V information. Hence, it can be said that U/V prediction is not always required. On the other hand, the hybrid
methods tend to degrade intelligibility slightly compared toSS, owing to several issues, such as the eﬀect of synthesis
by using vocoder and using 24-dimensional mel-cepstral coeﬃcients as spectral features.
These results suggest that the proposed hybrid approach to EL speech enhancement based on the continuous
F0 estimation and using the low-pass filter is eﬀective in significantly improving naturalness of EL speech while avoiding degradation in listenability that is often observed in theconventional VC-based enhancement method.

IEICE TRANS. INF. & SYST., VOL.E97–D, NO.6 JUNE 2014

1436

5.

Conclusion

In this paper, we have proposed a hybrid approach to electrolaryngeal (EL) speech enhancement based on spectral
subtraction for spectral parameter estimation and statisticalvoice conversion for excitation parameter prediction. To further improve the excitation features estimation, we implemented continuous F0 estimation and low-pass filtering aspart of the proposed approach. Moreover, we proposed amethod for avoiding U/V prediction errors causing degradation in intelligibility. As a result of an experimental evaluation, it has been demonstrated that the proposed approach iscapable of significantly improving naturalness of EL speech
while causing no adverse eﬀect such as the degradation in
intelligibility. Furthermore, U/V prediction is not always
required for EL speech enhancement.

Kitamura, “Speech parameter generation algorithms for HMMbased speech synthesis,” Proc. ICASSP, pp.1315–1318, June 2000.
[13] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Maximum likelihood voice conversion based on GMM with STRAIGHT mixed
excitation,” Proc. Interspeech, pp.2266–2269, Sept. 2006.
[14] K. Yu and S. Young, “Continuous F0 modelling for HMM based
statistical parametric speech synthesis,” IEEE Trans. Audio Speech
Language, vol.19, no.5, pp.1071–1079, July 2011.
[15] A. Sakurai and K. Hirose, “Detection of phrase boundaries inJapanese by low-pass filtering of fundamental frequency contours,”
Proc. ICSLP, vol.2, pp.817–820, Oct. 1996.
[16] H. Kawahara, I. Masuda-Katsuse, and A. Cheveigne, “Restructuring speech representations using a pitch-adaptive time-frequency
smoothing and an instantaneous-frequency-based F0 extraction:
Possible role of a repetitive structure in sounds,” SPECOM, vol.27,
no.3-4, pp.187–207, April 1999.

Acknowledgements
Part of this work was supported by JSPS KAKENHI Grant
Number 22680016.
References[1] H. Liu, Q. Zhao, M.X. Wan, and S.P. Wang, “Enhancement of electrolarynx speech based on auditory masking,” IEEE Trans. Biomed.
Eng., vol.53, no.5, pp.865–874, May 2006.
[2] K. Nakamura, T. Toda, H. Saruwatari, and K. Shikano, “Speakingaid systems using GMM-based voice conversion for electrolaryngeal
speech,” SPECOM, vol.54, no.1, pp.134–146, Jan. 2012.
[3] D. Doi, T. Toda, K. Nakamura, H. Saruwatari, and K. Shikano,“Alaryngeal speech enhancement based on one-to-many eigenvoice
conversion,” IEEE Trans. Audio. Speech Language, vol.22, no.1,
pp.172–183, Jan. 2014.
[4] S.F. Boll, “Suppression of acoustic noise in speech using spectral
subtraction,” IEEE Trans. Acoust. Speech Signal Process., vol.27,
no.2, pp.113–120, April 1979.
[5] Y. Stylianou, O. Cappe, and E. Moulines, “Continuous probabilistictransform for voice conversion,” IEEE Trans. Speech Audio Process., vol.6, no.2, pp.131–142, March 1998.
[6] T. Toda, A.W. Black, and K. Tokuda, “Voice conversion based on
maximum likelihood estimation of spectral parameter trajectory,”
IEEE Trans. Audio Speech Language, vol.15, no.8, pp.2222–2235,
Nov. 2007.
[7] H.R. Sharifzadeh, I.V. McLoughlin, and F. Ahmadi, “Reconstruction of normal sounding speech for laryngectomy patients througha modified CELP codec,” IEEE Trans. Biomed. Eng., vol.57, no.10,
pp.2448–2458, Oct. 2010.
[8] B.L. Sim, Y.C. Tong, J.S. Chang, and C.T. Tan, “A parametricformulation of the generalized spectral subtraction method,” IEEE
Trans. Speech Audio Process., vol.6, no.4, pp.328–337, July 1998.
[9] H. Kawahara, J. Estill, and O. Fujimura, “Aperiodicity extraction
and control using mixed mode excitation and group delay manipulation for a high quality speech analysis, modification and system
STRAIGHT,” Proc. 2nd MAVEBA, Sept. 2001.
[10] T. Toda, M. Nakagiri, and K. Shikano, “Statistical voice conversion techniques for body-conducted unvoiced speech enhancement,”
IEEE Trans. Audio Speech Language, vol.20, no.9, pp.2505–2517,
Nov. 2012.
[11] A. Kain and M.W. Macon, “Spectral voice conversion for text-tospeech synthesis,” Proc. ICASSP, pp.285–288, May 1998.
[12] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T.

Kou Tanaka
graduated from the Department of Mathematics and Informatics, Facultyof Human Development, Kobe University inJapan in 2012. He is currently in the master’scourse at the Graduate School of Information
Science, Nara Institute of Science and Technology (NAIST) in Japan. He is a student memberof ISCA, and ASJ.

TANAKA et al.: A HYBRID APPROACH TO ELECTROLARYNGEAL SPEECH ENHANCEMENT

1437

Tomoki Toda
was born in Aichi, Japan on
January 18, 1977. He earned his B.E. degreefrom Nagoya University, Aichi, Japan, in 1999
and his M.E. and D.E. degrees from the Graduate School of Information Science, NAIST,
Nara, Japan, in 2001 and 2003, respectively. He
was a Research Fellow of JSPS in the Graduate School of Engineering, Nagoya Institute ofTechnology, Aichi, Japan, from 2003 to 2005.
He was an Assistant Professor of the Graduate School of Information Science, NAIST from2005 to 2011, where he is currently an Associate Professor. He has also
been a Visting Researcher at the NICT, Kyoto, Japan, since May 2006.
From March 2001 to March 2003, he was an Intern Researcher at the ATR
Spoken Language Communication Research Laboratories, Kyoto, Japan,
and then he was a Visiting Researcher at the ATR until March 2006. He wasalso a Visiting Researcher at the Language Technologies Institute, CMU,
Pittsburgh, USA, from October 2003 to September 2004 and at the Department of Engineering, University of Cambridge, Cambridge, UK, fromMarch to August 2008. His research interests include statistical approaches
to speech processing such as voice transformation, speech synthesis, speech
analysis, speech production, and speech recognition. He received the 18th
TELECOM System Technology Award for Students and the 23rd TELECOM System Technology Award from the TAF, the 2007 ISS Best Paper
Award and the 2010 ISS Young Researcher’s Award in Speech Field fromthe IEICE, the 10th Ericsson Young Scientist Award from Nippon Ericsson K.K., the 4th Itakura Prize Innovative Young Researcher Award andthe 26th Awaya Prize Young Researcher Award from the ASJ, the 2009
Young Author Best Paper Award from the IEEE SPS, the Best Paper Award(Short Paper in Regular Session Category) from APSIPA ASC 2012, the2012 Kiyasu Special Industrial Achievement Award from the IPSJ, and the2013 Best Paper Award (Speech Communication Journal) from EURASIPISCA. He was a member of the Speech and Language Technical Committeeof the IEEE SPS from 2007 to 2009. He is a member of IEEE, ISCA, IPSJ,
and ASJ.

Graham Neubig
received his B.E.
from University of Illinois, Urbana-Champaign,
U.S.A, in 2005, and his M.E. and Ph.D. in informatics from Kyoto University, Kyoto, Japan in2010 and 2012 respectively. He is currently an
assistant professor at the Nara Institute of Science an Technology, Nara, Japan. His researchinterests include speech and natural language
processing, with a focus on machine learning
approaches for applications such as machinetranslation, speech recognition, and spoken dialog.

Sakriani Sakti
received her B.E degree
in Informatics (cumlaude) from Bandung Institute of Technology, Indonesia, in 1999. In2000,
she received “DAAD-Siemens Program Asia
21st Century” Award to study in Communication Technology, University of Ulm, Germany,and received her MSc degree in 2002. During
her thesis work, she worked with Speech Understanding Department, DaimlerChrysler Research Center, Ulm, Germany. Between 2003–
2009, she worked as a researcher at ATR SLC
Labs, Japan, and during 2006–2011, she worked as an expert researcher
at NICT SLC Groups, Japan. While working with ATR-NICT, Japan, she
continued her study (2005–2008) with Dialog Systems Group University
of Ulm, Germany, and received her PhD degree in 2008. She actively
involved in collaboration activities such as Asian Pacific Telecommunity
Project (2003–2007), A-STAR and U-STAR (2006–2011). She also served
as a visiting professor of Computer Science Department, University of Indonesia (UI) in 2009–2011. Currently, she is an assistant professor of theAugmented Human Communication Lab, NAIST, Japan. She is a memberof JNS, SFN, ASJ, ISCA, IEICE amd IEEE. Her research interests include
statistical pattern recognition, speech recognition, spoken language translation, cognitive communication, and graphical modeling framework.

Satoshi Nakamura
received his B.S. from
Kyoto Institute of Technology in 1981 and Ph.D.
from Kyoto University in 1992. He was a director of ATR Spoken Language Communication Research Laboratories in 2000–2008, and avice president of ATR in 2007–2008. He wasa director general of Keihanna Research Laboratories, National Institute of Information andCommunications Technology, Japan in 2009–2010. He is currently a professor and a director of Augmented Human Communication laboratory, Graduate School of Information Science at Nara Institute of Science and Technology. He is interested in modeling and systems of spoken
dialog system, speech-to-speech translation. He is one of the leaders ofspeech-to-speech translation research projects including C-STAR, IWSLT
and A-STAR. He headed the world first network-based commercial speechto-speech translation service for 3-G mobile phones in 2007 and VoiceTra
project for iPhone in 2010. He received LREC Antonio Zampoli Award,
the Commendation for Science and Technology by the Ministry of Scienceand Technology in Japan. He is an elected board member of ISCA, International Speech Communication Association, and an elected member ofIEEE SPS, speech and language TC.