AN EVALUATION OF ALARYNGEAL SPEECH ENHANCEMENT METHODS BASED ON
VOICE CONVERSION TECHNIQUES
Hironori Doi, Keigo Nakamura, Tomoki Toda, Hiroshi Saruwatari, and Kiyohiro Shikano
Graduate School of Information Science, Nara Institute of Science and Technology, Nara, 630-0192 Japan
E-mail: {hironori-d, kei-naka, tomoki, sawatari, shikano}@is.naist.jp
ABSTRACTIn this study, we evaluate our proposed methods for enhancing
alaryngeal speech based on statistical voice conversion techniques.
Voice conversion based on a Gaussian mixture model has been applied to the conversion of alaryngeal speech into normal speech(AL-to-Speech). Moreover, one-to-many eigenvoice conversion(EVC) has also been applied to AL-to-Speech to enable the recoveryof the original voice quality of laryngectomees even if only one
arbitrary utterance of the original voice is available. VC/EVC-based
AL-to-Speech systems have been developed for several types ofalaryngeal speech, such as esophageal speech (ES), electrolaryngeal
speech (EL), and body-conducted silent electrolaryngeal speech(silent EL). These proposed systems are compared with each otherfrom various perspectives. The experimental results demonstratethat our proposed systems yield signiﬁcant enhancement effects oneach type of alaryngeal speech.
Index Terms— alaryngeal speech, speech enhancement, voice
conversion, eigenvoice conversion, performance evaluations
1. INTRODUCTION
Laryngectomees who have undergone total laryngectomy due toan accident or laryngeal cancer cannot produce speech sounds in aconventional manner because their vocal folds have been removed.
Therefore, they require an alternative speaking method to produce
speech sounds using sound sources generated in a special mannerwithout vibrating their vocal folds. The produced speech is called
alaryngeal speech.
There are various methods of producing alaryngeal speech. In
this study, we focus on the three types of alaryngeal speech shownin Fig. 1: esophageal speech (ES), electrolaryngeal speech (EL), andbody-conducted silent electrolaryngeal speech (silent EL). ES and
EL are the most popular types of alaryngeal speech in Japan. In
ES, alternative excitation sounds are generated by releasing gases
from or through the esophagus. Thus, ES can be produced without
any equipment. However, it is difﬁcult to learn the skills to produce ES. On the other hand, EL is produced using an electrolarynx, a medical device for mechanically generating the sound source
signals that are conducted into the oral cavity from the skin on thelower jaw. It is much easier to learn how to speak using the electrolarynx than to learn how to produce ES. However, because theelectrolarynx needs to generate sufﬁcient loud-sound source signals
to make the produced speech sufﬁciently audible, sound source signals are easily emitted outside, disturbing speech communication.
To resolve this issue, a speaking method for silent EL has been proposed [1]. A new sound source unit is used to generate less audible
sound source signals. Since the produced speech also becomes lessaudible, it is detected with a non audible murmur (NAM) microphone, which is a body-conductive microphone capable of detecting extremely soft speech from the neck below the ear. These three
This work was supported in part by MIC SCOPE. The authors are grateful to Prof. Hideki Kawahara of Wakayama University, Japan for permission
to use the STRAIGHT analysis-synthesis method.

978-1-4577-0539-7/11/$26.00 ©2011 IEEE

Fig. 1. Methods of producing three types of alaryngeal speech (ES,EL, and silent EL).
types of alaryngeal speech allow laryngectomees to produce speechsounds again. However, their sound quality and intelligibility areseverely degraded compared with those of normal speech utteredby non-laryngectomees. Moreover, alaryngeal speech sounds areof similar quality regardless of the speaker. This problem is called
degradation of speaker individuality.
As one of the techniques for effectively changing voice quality
while keeping linguistic contents unchanged, statistical voice conversion (VC) has been studied for around two decades [2, 3, 4]. In
particular, speaker conversion based on a Gaussian mixture model(GMM) [3] has been widely studied and its performance has beensigniﬁcantly improved [4]. A GMM of the joint probability densityof acoustic features between a source speaker’s voice and a target
speaker’s voice is trained in advance using parallel data consistingof dozens of utterance pairs of the source and target speakers. The
trained GMM is capable of converting the acoustic features of thesource speech to those of the target speech in a probabilistic manner.
Moreover, to make the training process more ﬂexible, one-to-many
eigenvoice conversion (EVC) [5] has been proposed as a method of
converting a single speaker’s voice into an arbitrary speaker’s voice.
This method enables us to control the speaker individuality of theconverted speech by manipulating a small number of parameters or
by automatically adjusting them to an arbitrary target speaker using
only a few target speech samples as adaptation data.
To effectively enhance alaryngeal speech, we have proposedan alaryngeal speech enhancement method that converts alaryngeal
speech into normal speech using VC techniques. The proposed
method is called alaryngeal speech-to-speech (AL-to-Speech) [6, 7].
AL-to-Speech yields signiﬁcant improvements in speech quality
since the converted speech is basically generated according to thestatistical properties of the acoustic features of normal speech.
Moreover, to enable ﬂexible control of the converted voice quality,
one-to-many EVC has also been applied to the conversion of ALto-Speech [6]. EVC-based AL-to-Speech allows laryngectomees torecover their original natural voice quality even if only one arbitraryutterance of their natural speech is available. Using these techniques,
we have preliminarily developed AL-to-Speech systems for ES, EL,
and silent EL [8].
In this paper, we describe our proposed AL-to-Speech systems

5136

ICASSP 2011

based on VC/EVC for individual types of alaryngeal speech andevaluate their effectiveness. A comparison between VC and EVC
in AL-to-Speech and a comparison among individual AL-to-Speechsystems are conducted from various perspectives. The enhancement
effects of the individual systems are demonstrated from the experimental results.
2. ALARYNGEAL SPEECH
2.1. Esophageal speech (ES)ES sounds more natural than other types of alaryngeal speech. Additionally, a speaker skilled in producing ES can control prosody using
residual organs. However, a spectral envelope varies more unstably than that in normal speech. Moreover, speciﬁc unnatural sounds
caused at producing the excitation sounds are often observed. Even
if we can perceive pitch information in ES, it is difﬁcult to directly
extract F0 patterns corresponding to pitch patterns from ES because
excitation signals are less periodic. We have found that pitch information is also perceived in an ES sample resynthesized from a
mel-cepstrum sequence including power coefﬁcients and noise excitation. Therefore, it is expected that pitch information of ES is
included in a spectral envelope.
2.2. Electrolaryngeal speech (EL)EL sounds mechanical owing to artiﬁcial excitation signals. Although a spectral envelope stably varies according to each phoneme,
it is distorted by the sound source signals leaked from the electrolarynx. The electrolarynx used in this paper generates sound source
signals with almost constant F0 values and a high periodicity. Excitation parameters such as F0 and aperiodic components are easily
extracted from EL but are less informative since they capture only
the acoustic characteristics of the artiﬁcial excitation signals.
2.3. Body-conducted silent electrolaryngeal speech (silent EL)
Silent EL sounds much more unnatural than EL owing to its lowerpowered sound source signals and body conduction. It basically has
similar acoustic characteristics to EL except that 1) the signal-tonoise ratio of silent EL is much lower than that of EL and 2) a severe
attenuation of high-frequency components over 3 or 4 kHz is inducedby the lack of radiation characteristics from the lips and by the effectof the low-pass characteristics of the soft tissue.
3. AL-TO-SPEECH
In AL-to-Speech, the spectrum, aperiodic components, and F0 of
the target normal speech are independently estimated using GMMsor eigenvoice GMMs (EV-GMMs) from the spectrum of alaryngeal
speech. To eliminate unstable ﬂuctuations observed in a spectrum sequence of alaryngeal speech and to compensate for a spectral structure collapsed by the production mechanisms of individual alaryngeal speech, a segment feature vector extracted from multiple framesaround a current analyzed frame is used as the input feature. We use
three joint static and dynamic feature vectors of the spectrum, aperiodic components, and F0 extracted from target normal speech as theoutput features.
In this section, we describe AL-to-Speech based on one-to-many
EVC. This method entails a training, adaptation, and conversion process. The details of AL-to-Speech based on VC are shown in ref. [6].
3.1. Training process
As conversion models for estimating spectrum and aperiodic components of normal speech from the spectral segment of alaryngeal
speech, EV-GMMs are trained using multiple parallel data sets consisting of utterance pairs of a laryngectomee and many prestored target speakers. Let us assume a source spectral segment feature vector,
Xt , and a target joint static and dynamic feature vector, Yt , at framet. The EV-GMM models the joint probability density of the source
and target feature vectors as;
P (X t , Y t |λ(EV ) , w) 
(X,Y ))(1)
αm N [X (w), Σ(X,Yt , Y t ] ; μm
m
m=1




(XX)
(XY )
Σm
Σm
μ(X)
(X,Y )(X,Y )
m
, Σm
(w) =
=
μm
(Y X)
(Y Y )Am w + bm
ΣmΣm(2)
where w = [w(1), · · · , w(J)] is the target-speaker-dependent
weight vector for controlling target voice quality.  denotes thetransposition of the vector. λ(EV ) is a canonical EV-GMM parameter set consisting of the weight αm , the mean vector μ(X)m , the(X,Y ), the bias vector bm , and the eigenveccovariance matrix Σmtors Am = [am (1), · · · , am (J)] for the mth mixture component,
where the number of eigenvectors is J.
The EV-GMM is adapted to a new target speaker by adjusting
the weight vector so that the marginal likelihood for given target
speech features is maximized [5]. This adaptation process is effective if speaker-dependent characteristics are well captured byshort-term features, such as spectrum and aperiodic components.
On the other hand, it is essentially difﬁcult to control speakerdependent characteristics captured by long-term features, such as F0
patterns. Therefore, instead of the EV-GMM, a well-trained speakerdependent GMM is used to estimate F0 patterns from the spectral
segment sequence of alaryngeal speech. In AL-to-Speech for ES,
to develop the GMM for estimating F0 patterns corresponding to
the perceived pitch information of ES, we use F0 values extracted
from normal speech uttered by a non-laryngectomee as an imitatingprosody of ES in the training as the output features [6]. To developa GMM for F0 estimation in EL and silent EL, speaker-dependent
GMMs are separately trained for all prestored target speakers. Then,the GMM achieving the highest F0 estimation accuracy is manually
selected.
3.2. Adaptation and conversion processes
Assuming that a few speech samples uttered by laryngectomees
before undergoing total laryngectomy are available as adaptation
data, the EV-GMM is ﬂexibly adapted to the target voice quality byautomatically determining the weight vector in a text-independent
manner [5]. The weight vectors of the EV-GMMs for the spectral
and aperiodic estimations are independently estimated using thespectral features and the aperiodic components extracted from thegiven target speech samples. The converted spectral feature vectors
and aperiodic components are independently estimated using theadapted EV-GMMs. On the other hand, in the F0 estimation, theglobal speaker-dependent characteristics of F0 patterns are simply
controlled. A log-scaled F0 sequence is ﬁrst estimated with theselected speaker-dependent GMM, and then further converted sothat its mean μx and standard deviation σx are equal to those of theadaptation speech data, μy and σy , as follows:σy
log yt =(log xt − μx ) + μy ,(3)σxwhere xt and yt denote the F0 value estimated with the GMM
and the converted F0 value at frame t, respectively. The maximum likelihood estimation method considering not only the explicit
relationship between static and dynamic features, but also global
variance [4], is adopted in the estimation of the converted features.
=M

4. EXPERIMENTAL EVALUATION
4.1. Experimental conditions
We recorded 50 phoneme-balanced sentences of ES uttered by one
Japanese male laryngectomee, those of EL and silent EL uttered by5137

another Japanese male laryngectomee, and those of normal speechuttered by each of 40 Japanese non-laryngectomees. The speech
data of 30 non-laryngectomees were used for training and those of
the other 10 non-laryngectomees were used as the target data forevaluation. From the 50 recorded sentences of each speaker, 40 wereused as the training or adaptation data and the remaining 10 wereused as the test data. The sampling frequency was set to 16 kHz.
The 0th through 24th mel-cepstral coefﬁcients were used as
spectral parameters. Mel-cepstrum analysis [9] was employed foralaryngeal speech and STRAIGHT analysis [10] was employed fornormal speech. The frame shift was 5 ms. To extract the spectral
segment feature of ES, current and ±8 frames were used for spectral
and aperiodic estimation and current and ±16 frames were used
for F0 estimation. For EL and silent EL, current and ±8 frames
were used for each parameter estimation. These frame lengths werepreliminarily optimized. As the source excitation features of normal
speech, we used log-scaled F0 values and aperiodic components fordesigning mixed excitation.
The EV-GMMs for spectral and aperiodic component estimation were trained for each type of alaryngeal speech. The numbersof eigenvectors and mixture components were set to 29 and 64 inevery EV-GMM, respectively. The EV-GMMs were adapted to thetarget speakers using 1, 2, 4, 8, 16, or 32 utterances of their normal
speech data. For AL-to-Speech based on VC, the GMMs for spectral and aperiodic estimation were trained using a parallel dataset
for each type of alaryngeal speech and normal speech of each target speaker. The number of training utterance pairs was set to 1, 2,
4, 8, 16 or 32. The number of mixture components was optimized
manually depending on the training data size. Individual speakerdependent GMMs for F0 estimation were trained for all the 40 nonlaryngectomees. The GMM yielding the most natural F0 pattern wasthen selected by listening to the converted speech. The same F0 estimation process was performed for the EVC-based AL-to-Speech
and VC-based AL-to-Speech.
4.2. Objective evaluation
We evaluated the effectiveness of AL-to-Speech based on EVC/VC
for each type of alaryngeal speech with root mean square error(RMSE) on aperiodic components. The result of mel-cepstral distortion is shown in ref. [8]. Figure 2 shows RMSE on aperiodic
components as a function of the number of adaptation utterancesused in EVC or of utterance pairs used in VC. EVC shows a significantly smaller RMSE than VC in each type of alaryngeal speechenhancement when the amount of the target normal speech data
is small. Even if only one arbitrary utterance of the target normal
speech is available in EVC, its conversion performance is almost
equivalent to or better than that of VC using 16 parallel utterance
pairs. It is also observed that ES yields the best conversion accuracy
and silent EL yields the worst among the three types of alaryngeal
speech. Note that similar results have been observed in mel-cepstral
distortion [8].
We also evaluated the F0 estimation accuracy in AL-to-Speechfor each type of alaryngeal speech using F0 correlation coefﬁcientand Unvoiced/Voiced (U/V) error between converted speech and target normal speech. To demonstrate the F0 estimation accuracy forvarious speakers in AL-to-Speech, the results calculated using individual speaker-dependent GMMs for the 40 non-laryngectomees areshown in Table 1. For ES, the results for another non-laryngectomee
who uttered normal speech so that its pitch sounded similar to thatof ES are also shown as ”ES pitch.” ES yields the best estimation accuracy among the three types of alaryngeal speech. Additionally, theestimation accuracy is signiﬁcantly improved using the GMM developed with the normal speech, the F0 patterns of which correspond

Fig. 2. RMSE on aperiodic components as a function of the numberof utterances of target normal speech (i.e., utterance pairs in VC or
adaptation utterances in EVC).
Table 1. F0 estimation accuracies for various target speakers using
corresponding target-speaker-dependent GMMs
Correlation
U/V error [%]ES
0.58
12.39 (V → U : 6.59, U → V : 5.80)
EL
0.40
13.20 (V → U : 4.92, U → V : 8.28)
Silent EL
0.42
14.02 (V → U : 6.89, U → V : 7.13)
ES pitch
0.68
8.36 (V → U : 4.30, U → V : 4.05)
well to the pitch patterns of ES.
The ﬁnal results for the 10 target non-laryngectomees from thetest data are shown in Table 2. The GMM for ”ES pitch” was used in
ES enhancement, and manually selected speaker-dependent GMMs
were used in the ES/silent EL enhancement. Namely, the speaker
used in the model training is different from the target speakers. It
is observed that, for EL and silent EL, the estimation accuracy ofthe selected GMMs is higher than that of various speaker-dependent
GMMs shown in Table 1, even though a speaker different from thetarget speakers is used in the training. To generate a natural F0 pattern in AL-to-Speech, it is useful to select an optimum speaker for
training rather than to directly use the same speaker as the actual target speaker since the F0 estimation accuracy largely varies among
different speakers. It is also observed that ES enhancement yields
better F0 correlation than the others.
4.3. Subjective evaluation
We conducted opinion tests of speech quality and intelligibility. In
the opinion test of intelligibility, 8 listeners evaluated 9 types ofspeech including original alaryngeal speech and converted speechwith AL-to-Speech based on VC/EVC in ES, EL, and silent EL. The
VC-based AL-to-Speech used 32 utterance pairs for GMM training.
On the other hand, only one utterance was used as adaptation data for
the EVC-based AL-to-Speech. Each listener evaluated 135 speechsamples. The experimental conditions for the opinion test of speech
quality are shown in ref. [8]. We also conducted a preference test
to evaluate speaker individuality. In the preference test, 6 listeners
evaluated 6 types of speech consisting of converted speech by the
VC/EVC-based AL-to-Speech in EL, ES, and silent EL. The training data used in VC and the adaptation data used in EVC were thesame as those used in the opinion tests.
Figures 3 and 4 show the results of the opinion tests of speech
quality and intelligibility, respectively. All the AL-to-Speech methods yield signiﬁcant improvements in speech quality compared withthat of the original alaryngeal speech. The speech quality of the enhanced silent EL is lower than that of the enhanced ES and enhanced
EL but it is signiﬁcantly higher than that of each type of original ala-

5138

Table 2. F0 estimation accuracies for actual target speakers in evaluation using well-trained speaker-dependent GMMs
ES pitch
0.62 13.88 (V → U : 10.70, U → V : 3.18)
EL
0.51
12.05 (V → U : 7.13, U → V : 4.92)
Silent EL 0.45
13.78 (V → U : 8.92, U → V : 4.86)

Fig. 4. Result of opinion test of intelligibility

Fig. 3. Result of opinion test of speech quality. ”Org”, ”VC”, and”EVC” show original alaryngeal speech, converted speech by AL-toSpeech based on VC trained with 32 utterance pairs, and converted
speech by AL-to-Speech based on one-to-many EVC adapted with
one utterance of target speech, respectively.
ryngeal speech. The intelligibilities of ES and silent EL are also
improved by AL-to-Speech. On the other hand, the intelligibility of
EL slightly degrades from that of the original EL by AL-to-Speech,
as observed in our previous work [7]. The speech quality and intelligibility enhanced by the EVC-based AL-to-Speech are almost
equivalent to those enhanced by the VC-based AL-to-Speech. Notethat the EVC-based method requires only one arbitrary utterance ofthe target normal speech whereas the VC-based method requires 32
utterance pairs of alaryngeal speech and the target normal speech.
Figure 5 shows the result of the preference test. We can observe the same tendency as that in Fig. 2. Enhanced ES yields thebest speaker individuality and enhanced silent EL yields the worst
among the three types of alaryngeal speech. We can observe that theVC-based methods slightly outperform the EVC-based methods in
ES and EL. This tendency depends on the amount of available parallel data used for GMM training in VC-based methods, as shown in
Fig. 2.
5. CONCLUSIONSIn this paper, we evaluated our proposed statistical enhancement
methods based on voice conversion techniques (AL-to-Speech) for
three types of alaryngeal speech: esophageal speech (ES), electrolaryngeal speech (EL), and body-conducted silent electrolaryngeal
speech (silent EL). The experimental results suggested that 1) the
proposed methods signiﬁcantly improve the speech quality of eachtype of alaryngeal speech, 2) the proposed methods also improve theintelligibilities of ES and silent EL, 3) AL-to-Speech based on eigenvoice conversion (EVC) is capable of effectively adjusting the voice
quality of enhanced speech to the target voice quality using only one
arbitrary utterance of the target voice, and 4) AL-to-Speech for ES
is the best in terms of speech quality, intelligibility, and speaker individuality.
6. REFERENCES[1] K. Nakamura, T. Toda, H. Saruwatari, and K. Shikano, “Evaluation of
extremely small sound source signals used in speaking-aid system with

5139

Fig. 5. Result of preference test of speaker individuality
statistical voice conversion,” IEICE Trans. Inf. and Syst., vol. E93-D,
no. 7, pp. 1909–1917, July 2010.
[2] M. Abe, S. Nakamura, K. Shikano, and H. Kuwabara, “Voice conversion through vector quantization,” J. Acoust. Soc. Jpn. (E), vol. 11, no.
2, pp. 71–76, 1990.
[3] Y. Stylianou, O. Cappe, and E. Moulines, “Continuous probabilistictransform for voice conversion,” IEEE Trans. SAP, vol. 6, no. 2, pp.
131–142, 1998.
[4] T. Toda, A. W. Black, and K. Tokuda, “Voice conversion based on
maximum likelihood estimation of spectral parameter trajectory,” IEEE
Trans. ASLP, vol. 15, no. 8, pp. 2222–2235, Nov. 2007.
[5] T. Toda, Y. Ohtani, and K. Shikano, “Eigenvoice conversion based on
gaussian mixture model,” Interspeech, pp. 2446–2449, Sept. 2006.
[6] H. Doi, K. Nakamura, T. Toda, H. Saruwatari, and K. Shikano,“Esophageal speech enhancement based on statistical voice conversion
with gaussian mixture models,” IEICE Trans. Inf. and Syst., vol. E93D, no. 9, pp. 2472–2482, Sept. 2010.
[7] K. Nakamura, T. Toda, H. Saruwatari, and K. Shikano, “The use of
air-pressure sensor in electrolaryngeal speech enhancement based on
statistical voice conversion,” Interspeech, pp. 1628–1631, Sept. 2010.
[8] H. Doi, K. Nakamura, T. Toda, H. Saruwatari, and K. Shikano,“Speaking-aid system based on one-to-many eigenvoice conversion for
total laryngectomees,” APSIPA ASC, pp. 498–501, Dec. 2010.
[9] K. Tokuda, T. Kobayashi, T. Masuko, and S. Imai, “Mel-generalized
cepstral analysis – a uniﬁed approach to speech spectral estimation,”ICSLP, pp. 1043–1045, Sept 1994.
[10] H. Kawahara, I. Masuda-Katsuse, and A. Cheveigne, “Restructuring
speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based f0 extraction: Possible roleof a repetitive structure in sounds,” Speech Communication, vol. 27,
no. 3-4, pp. 187–207, 1999.