172

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 22, NO. 1, JANUARY 2014

Alaryngeal Speech Enhancement Based on
One-to-Many Eigenvoice Conversion
Hironori Doi, Tomoki Toda, Member, IEEE, Keigo Nakamura, Hiroshi Saruwatari, Member, IEEE, andKiyohiro Shikano, Fellow, IEEE

Abstract—In this paper, we present novel speaking-aid systems
based on one-to-many eigenvoice conversion (EVC) to enhance
three types of alaryngeal speech: esophageal speech, electrolaryngeal speech, and body-conducted silent electrolaryngeal speech.
Although alaryngeal speech allows laryngectomees to utter speechsounds, it suffers from the lack of speech quality and speakerindividuality. To improve the speech quality of alaryngeal speech,
alaryngeal-speech-to-speech (AL-to-Speech) methods based on
statistical voice conversion have been proposed. In this paper,
one-to-many EVC capable of flexibly controlling the converted
voice quality by adapting the conversion model to given target
natural voices is further implemented for the AL-to-Speechmethods to effectively recover speaker individuality of each typeof alaryngeal speech. These proposed systems are compared witheach other from various perspectives. The experimental results
demonstrate that our proposed systems are capable of effectively
addressing the issues of alaryngeal speech, e.g., yielding significant
improvements in speech quality of each type of alaryngeal speech.
Index Terms—Alaryngeal speech, eigenvoice conversion, laryngectomees, speech enhancement, voice conversion.

I. INTRODUCTIONPATIENTS who suffer from laryngeal cancer require total
laryngectomy, which is a surgical operation to remove
the larynx and tissues around the larynx such as the vocal
folds. People who have undergone total laryngectomy, called
laryngectomees, cannot speak in the usual manner owing to theremoval of their vocal folds. Because speech is one of the mostpopular methods of human communication, laryngectomees
experience inconvenience in many situations of their daily life.
Therefore, they desire to be able to speak using medical devices
or following rehabilitation in order to reintegrate into their individual, social, and regular activities. To accomplish their wish,
alternative speaking methods to produce speech sounds using
residual organs or medical devices instead of vocal cords havebeen used. Speech sounds generated by alternative speaking

Manuscript received March 27, 2013; revised July 10, 2013; accepted October 03, 2013. Date of publication October 23, 2013; date of current version
November 22, 2013. This work was supported in part by MIC SCOPE and
MEXT Grant-in-Aid for Young Scientists (A) and in part by a JSPS Research
Fellowships for Young Scientists. The associate editor coordinating the reviewof this manuscript and approving it for publication was Prof. Mark J. F. Gales.
H. Doi, T. Toda, H. Saruwatari, and K. Shikano are with the Graduate Schoolof Information Science, Nara Institute of Science and Technology, Ikoma,
Nara 630-0192, Japan (e-mail: hironori-d@is.naist.jp; tomoki@is.naist.jp;
sawatari@is.naist.jp; shikano@is.naist.jp).
K. Nakamura was with the Graduate School of Information Science, Nara
Institute of Science and Technology, Ikoma, Nara 630-0192, Japan. He is now
with Rakuten, Inc., Tokyo 140-0002, Japan.
Digital Object Identifier 10.1109/TASLP.2013.2286917

Fig. 1. Alternative speaking methods for laryngectomees: (a) to produce
esophageal (ES) speech, (b) to produce electrolaryngeal (EL) speech, and (c) to
produce silent electrolaryngeal (silent EL) speech.

methods without vocal fold vibration are called alaryngeal
speech.
There are various alternative speaking methods. In thispaper, three speaking methods shown in Fig. 1 are focusedon. Among them, the speaking methods for esophageal speech(ES speech) and electrolaryngeal speech (EL speech) are themost popular methods in Japan. The speaking method for ES
speech is one of the alternative speaking methods that generate
alaryngeal speech with residual organs. ES speech is generated
by modulating alternative excitation sounds that are produced
by releasing gases from or through the esophagus by articulatory movement. This speaking method allows laryngectomees
to speak without any equipment and ES speech sounds morenatural than the other types of alaryngeal speech such as EL
speech, but its sound quality is not comparable to normal
speech uttered by non laryngectomees. Although it generally
takes a long time to learn the speaking method for ES speech,
support for learning it is provided by many volunteers in Japan.
The speaking method for EL speech is one of the mostpopular alternative speaking methods using medical devices.
Alternative excitation sounds are produced using an electrolarynx, which is a medical device to mechanically generate
the sound source signals. The generated sound source signals
are conducted as alternative excitation sounds into the oralcavity from the skin on the lower jaw. Then, the alternative
excitation sounds are articulated to produce EL speech sounds.
It is much easier to learn how to speak using the electrolarynx
than to learn how to produce ES speech. Moreover, users needless physical power to produce EL speech compared withother alaryngeal speech, such as ES speech. However, the EL
speech sound is mechanical and artificial because the generated
fundamental frequency ( ) contour is totally unnatural owingof theto the pre-defined frequency of the vibration (i.e.,

2329-9290 © 2013 IEEE

DOI et al.: ALARYNGEAL SPEECH ENHANCEMENT BASED ON ONE-TO-MANY EVC

sound source signals). Additionally, because the electrolarynx
needs to generate sufficiently loud sound source signals to
make the produced EL speech sufficiently audible, the soundsource signals are readily emitted outside, disturbing speechcommunication.
To resolve the issue of the emitted sound source signals inthe speaking method for EL speech, a new speaking method forsilent EL speech has been proposed [1]. A new sound source
unit is used to generate less audible sound source signals. Since
the produced speech also becomes less audible, it is detectedwith a nonaudible murmur (NAM) microphone [2], which isa body-conductive microphone capable of detecting extremely
soft speech from the neck below the ear. The detected speech
signals are presented outside as silent EL speech while keeping
the external sound source signals sufficiently silent.
Although these three types of alaryngeal speech allow laryngectomees to speak again, their sound quality and intelligibility are severely degraded compared with those of normal
speech. Moreover, alaryngeal speech sounds are of similar voice
quality1 regardless of the speaker differences because the production mechanism of the sound source signals in each type ofalaryngeal speech strongly affects the voice quality of the produced speech. Consequently, the alaryngeal speech suffers fromthe degradation of speaker individuality.
Several attempts at improving alaryngeal speech quality havebeen carried out. A new electrolarynx using an air pressure
sensor has been developed to controlof the sound source
signals by an expiratory pressure. Although it is not easy toaccurately control
by adjusting the expiratory pressure, thisdevice makes it possible for laryngectomees to produce morenaturally sounding EL speech, theof which effectively
varies over an utterance [3]. One weakness of this device isthat both hands are needed to hold an electrolarynx and airpressure sensor while speaking. Moreover, it is still difficult tomechanically generate sound source signals similar to thosenaturally generated by vocal fold vibrations. Consequently, theproduced EL speech quality is still different from the natural
voices produced by non laryngectomees.
As another attempt, speech enhancement methods based on
the modifications of acoustic features of ES speech using signal
processing, such as comb filtering [4], smoothing of acoustic
parameters [5], formant manipulation [6], and noise reduction
based on auditory masking [7] have been proposed. Although
they are useful in the alaryngeal speech enhancement, quality
improvements are still limited since the acoustic features of alaryngeal speech exhibit quite different properties from those of
normal speech, and therefore, it is basically difficult to compensate for those acoustic differences using such a simple modification process.
Recently, statistical approaches to alaryngeal speech enhancement have been proposed [8], [9] to convert alaryngeal
speech into target normal speech while keeping linguistic information unchanged. The statistical enhancement framework
consists of training and conversion processes. In the training
process, a conversion function from acoustic features of alaryn1In this paper, the term “voice quality” is used to represent speech characteristics on speaker individuality affected by both glottal excitation parametersand vocal tract spectral parameters.

173

geal speech into those of target normal speech is modeled using
training data including utterance pairs of alaryngeal speech andnormal speech. In the conversion process, any utterance of alaryngeal speech is converted to that of target normal speech on
the basis of the conversion function. This data-driven approachis capable of more complicated acoustic modifications to compensate for the large acoustic differences between alaryngeal
speech and normal speech. As typical conventional methods, acodebook mapping method [10] and a probabilistic conversion
method based on Gaussian mixture models (GMMs) [11] havebeen applied to alaryngeal speech enhancement [8], [9]. The
GMM-based conversion method is one of the most popular
voice conversion methods. It is well defined mathematicallyand its conversion performance is relatively high. It has beenreported that the alaryngeal speech enhancement method basedon the GMM-based voice conversion, which is called Alaryngeal-Speech-to-Speech (AL-to-Speech), is highly effective for
improving the naturalness and intelligibility of individual typesof alaryngeal speech [1], [3], [9].
Although these conventional enhancement methods allow laryngectomees to speak in more natural voices than alaryngeal
speech, recovering speaker individuality is minimally considered. In fact, it is essentially difficult to flexibly control the voice
quality of enhanced alaryngeal speech by these methods. In thestatistical voice conversion approaches, it is possible to change
the converted voice quality using different target voices but it isnecessary to prepare training data consisting of utterance pairsof the alaryngeal speech and each target voice, which is very
laborious. To flexibly change the converted voice quality to recover speaker individuality or provide a unique voice for laryngectomees, one-to-many eigenvoice conversion (EVC) [12] hasbeen applied to ES speech enhancement (called ES-to-Speech)
in our previous work [13]. One-to-many EVC is a techniquefor converting a specific source speaker’s voice into an arbitrary target speaker’s voice. This method allows us to control
the speaker individuality of the converted speech by manipulating a small number of parameters or to flexibly adapt theconversion model to an arbitrary target speaker on the basis ofa small number of given target speech samples in a text-independent manner. ES-to-Speech based on EVC helps laryngectomees speak in their favorite voices or in their own voices thathave already been lost but a few recorded samples are available.
In this study, we develop AL-to-Speech systems capable offlexibly controlling the enhanced voice quality based on one-tomany EVC for not only ES speech but also EL speech and silent
EL speech. The effectiveness of the proposed AL-to-Speechsystems based on VC/EVC for the three types of alaryngeal
speech is evaluated from various perspectives. The features ofeach AL-to-Speech system are demonstrated through various
comparisons among the different AL-to-Speech systems. In thispaper, we present further details of the proposed systems, morediscussions, and more evaluations than those in our previous
work [9].
The paper is organized as follows. In Section II, the characteristics of three types of alaryngeal speech, namely, ES speech,
EL speech and silent EL speech are described. In Section III, thestatistical VC algorithm and a one-to-many EVC algorithm aredescribed. In Section IV, the proposed method of enhancing ala-

174

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 22, NO. 1, JANUARY 2014

ryngeal speech is discussed. In Section V, our proposed method
is experimentally evaluated. Finally, this paper is summarized
in Section VI.
II. ALARYNGEAL SPEECH
The larynx including the vocal fold has to be removed by laryngectomy, if the larynx has severe trouble such as cancer. Because the larynx prevents food from entering the trachea, thetrachea and the oral cavity connecting the esophagus are completely separated from each other when this function is lost by
total laryngectomy. Therefore, laryngectomees cannot generate
vocal fold vibrations nor expire air through the oral cavity. They
have to produce speech sounds in an alternative manner.
In this section, we describe the three types of alaryngeal
speech generated by an alternative speaking method for laryngectomees. Fig. 2 shows an example of speech waveforms,
spectrograms,
contours, and aperiodic components of(a) normal speech, (b) ES speech, (c) EL speech, and (d) silent
EL speech in the same sentence.
contours are extracted with STRAIGHT analysis [14] bymanually optimizing an
search range to reduce
extraction
errors as much as possible. Spectrograms and 5-band aperiodic
components (averaged on 0-1, 1-2, 2-4, 4-6, and 6-8 kHz frequency bands) [15] are also extracted with STRAIGHT analysis[16], [17].
A. Esophageal Speech (ES speech)
ES speech sounds more natural than the other types of alaryngeal speech. Although a speaker skilled in producing ES
speech can control prosody using residual organs, the produced
sound is constantly low in tone regardless the speaker. Moreover, specific unnatural sounds caused by producing the excitation sounds in the manner mentioned above are often observed. The difficulty of producing ES speech makes a spectralenvelope vary unstably but it still captures phoneme-dependent
acoustic characteristics and is capable of conveying linguistic
information. Aperiodic components are constantly noisy in all
frequency bands.
Even if we can perceive pitch information (i.e., perceptual
tone related to prosody) in ES speech, it is difficult to directly
extract
patterns corresponding to pitch information from ES
speech because the excitation signals are less periodic. This issimilar to pitch perception in a whispered voice, which is unvoiced speech. We have found that pitch information is also perceived in an ES speech sample resynthesized from a mel-cepstrum sequence including power coefficients and noise excitation [13]. Therefore, we expect that an acoustic cue of pitch information in ES speech is included in its spectral envelope andpower.
B. Electrolaryngeal Speech (EL Speech)
It is easier to stably speak using EL speech than using ES
speech because stable excitation signals are generated using
medical device. However, EL speech sounds mechanical owing
to artificial excitation signals. Although the spectral envelope
stably varies according to each phoneme, it is distorted by thesound source signals leaked from the electrolarynx. The electrolarynx used in this study generates sound source signals withalmost constantvalues and a high periodicity. Excitation parameters such as
and aperiodic components are easily extracted from EL speech but are less informative since they capture only the acoustic characteristics of the artificial excitation
signals.
C. Body-Conducted Silent Electrolaryngeal Speech (Silent
EL Speech)
Silent EL speech allows the user to speak without leaked excitation sounds even with an electrolarynx. However, silent EL
speech sounds much more unnatural than EL speech owing to
its lower-powered sound source signals and body conduction.
It basically has similar acoustic characteristics to EL speech except that (1) the signal-to-noise ratio of silent EL speech is much
lower than that of EL speech and (2) high-frequency components over 3 or 4 kHz are severely attenuated by the lack ofradiation characteristics from the lips and by the effect of the
low-pass characteristics of the soft tissue [18].
III. VOICE CONVERSION
A. Basic Voice Conversion (VC)
VC is the method that converts the source speaker’s voice
into the target speaker’s voice in a probabilistic manner. In thissection, we describe a conversion method based on maximum
likelihood estimation of speech parameter trajectories considering global variance (GV) [19] as one of the state-of-the-art
VC methods. This method consists of a training and a conversion process.
1) Training Process: Let us assume adimensional
input static feature vector
and adimensional output static feature vector
at frame , respectively, where denotes
transposition of the vector. As an input speech parameter vector,
we useto capture contextual features of source speech, suchas the joint feature vector of static and dynamic feature vectors
or the concatenated feature vector from multiple frames. Asan output speech feature vector, we use
consisting of static feature vector
and a dynamic feature
vector
.
By using a parallel training data set consisting of time-aligned
input and output parameter vectors,
determined by Dynamic
Time Warping (DTW), where
denotes the total number offrames, the joint probability density of the input and output parameter vectors is modeled with a GMM [20] as follows:(1)(2)
denotes a Gaussian distribution with a mean
where
vector and a covariance matrix . The mixture component
index is . The total number of mixture components is . A
parameter set of the GMM is , which consists of weights,
and full covariance matrices
for inmean vectors
consists
dividual mixture components. The mean vector

DOI et al.: ALARYNGEAL SPEECH ENHANCEMENT BASED ON ONE-TO-MANY EVC

175

Fig. 2. Example of acoustic features, i.e., waveforms, spectrograms,
contours, and aperiodic components of (a) normal speech, (b) ES speech, (c) EL speech,and (d) silent EL speech in the same sentence fragment /h o N sy o w a k o t o b a n o/. In aperiodic components, the solid line, coarse broken line, and fine broken
line represent averaged aperiodic components in low frequency band, middle frequency band, and high frequency band, respectively.

of an input mean vector
and an output mean vector.
consists of input and output covariThe covariance matrix
andand cross-covariance matrices
ance matrices
and.
The GV is defined as the variance of features over one utterance. To consider the GV in the conversion, the probability
density of the GVof the output static feature vectors
over an utterance is also modeled witha Gaussian distribution,(3)
where the GV

is calculated as(4)

A parameter set
consists of a mean vector
and a diagonal covariance matrix.
Conversion Process: Letandbe a time sequence of the
input and output feature vectors, respectively. The converted
static feature vector sequenceisdetermined by maximizing an objective function defined as a
product of the GV probability density function given by Eq. (3)and the conditional probability density function,
which is analytically derived from the joint probability density
given (1), as follows:

subject to(5)

is a window matrix to extend the static feature vector
wheresequence into the joint feature vector sequence of static anddynamic features [21]. The balance between
is controlled by the weight .

and

B. One-to-Many Eigenvoice Coversion (EVC)
We also describe one-to-many EVC [12] as a technique forflexibly controlling the voice quality of the converted speech.
This method consists of training, adaptation, and conversion
process.
Training Process: In one-to-many EVC, Eigenvoice GMM(EV-GMM) is used as a conversion model. The EV-GMM istrained using multiple parallel data sets consisting of a single
input speech data set and many output speech data sets including
various speakers’ voices. EV-GMM models the joint probability
density of the input and output parameter vectors as follows:

(6)
(7)
(8)
where
is a target-speaker-independent param,
,
,
, and
eter set of EV-GMM, i.e.,
for themixture component, whereandare a bias vector
and eigenvectors, respectively. -dimensional weightvectoris a target-speaker-dependent
parameters for controlling target speaker individuality. The
number of eigenvectors is .
Adaptation Process: The trained EV-GMM allows users tocontrol the converted voice quality by manipulating the weight176

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 22, NO. 1, JANUARY 2014

TABLE I
CHARACTERISTICS OF ACOUSTIC FEATURES OF ALARYNGEAL SPEECH

vector . If users have target speech data, the GMM for thesource speech and new target speech are flexibly built by automatically determining the weight vector using only a few
arbitrary utterances of the target speech in a text-independent
manner. The optimal weight vector
is estimated by maximizing the likelihood of the marginal distribution as follows
[22]:
(9)
is a time sequence of the target features for thewhereadaptation.
Conversion Process: In the conversion process, the user
gives a target-speaker-dependent parameter by entering data
manually or by estimating from target speech samples. The
adapted EV-GMM is generated with as shown in (7). Then,the converted speech is estimated in the same manner as basic
VC shown in (5).
IV. VOICE CONVERSION FROM ALARYNGEAL SPEECH TO
SPEECH (AL-TO-SPEECH)
To enhance the three types of alaryngeal speech, namely, ES
speech, EL speech, and silent EL speech, statistical VC approaches for converting alaryngeal speech into normal speech
have been proposed. In AL-to-Speech based on VC, the converted speech sounds similar to normal speech because the converted speech parameters are basically determined according
to the statistics extracted from the normal speech in a probabilistic manner. Furthermore, by applying one-to-many EVC to
AL-to-speech, AL-to-Speech allows users to flexibly control the
speech quality of the converted speech. In this section, we describe AL-to-Speech based on VC and one-to-many EVC forthe enhancement of the three types of alaryngeal speech.
A. Feature Extraction in AL-to-Speech
In AL-to-Speech, three types of acoustic feature of normal
speech, namely, spectrum, aperiodic components, and , areseparately estimated from the acoustic features of each typeof alaryngeal speech. Then, the estimated acoustic features areused in vocoding to generate converted speech. To estimate theacoustic features of normal speech by AL-to-Speech, we need todecide which acoustic features of each type of alaryngeal speech
are used as the input feature. However, we have little choice because most of the acoustic features of alaryngeal speech are lessinformative as mentioned in Section II. Table I shows characteristics of acoustic features of each types of alaryngeal speech.
Because the spectrum of ES speech is the only informative
acoustic feature of ES speech even if it unstably varies, we use
the spectrum of ES speech as an input feature to estimate thespectrum and aperiodic components of normal speech, whichsmoothly vary according to the phoneme. On the other hand,forestimation, we assume that an acoustic cue of the pitchof ES speech could be included in the spectrum with power as
mentioned in Section II. On the basis of this assumption, we also
use the spectrum of ES speech as an input feature to estimate
of normal speech. This is similar to the conventional estimation
methods offrom mel-frequency cepstral coefficients [23],[24] but this is an estimation process offrom the spectrumof unvoiced speech rather than voiced speech, and therefore,
this estimation process is much more difficult compared withthe conventional ones. Furthermore, although one reasonable
conversion process is to estimate
corresponding to the pitchof ES speech, it is not straightforward to prepare target values
in training process as such an
is difficult to extract from ES
speech. To address this issue, we record normal speech utteredby non laryngectomee so that its pitch sounds similar to thatof ES speech and use
extracted from the recorded normal
speech as the target in training. Namely, corresponding to thepitch of ES speech is approximated withof normal speechuttered by a different speaker.
Although the spectra of EL speech and silent EL speechchange according to the phoneme, it is significantly different
from those of normal speech.
values of EL speech and silent
EL speech are mechanically decided independent of utterance
content. Moreover, because an electrolarynx is driven during
an utterance, the aperiodic components of EL speech and silent
EL speech depending on only the mechanical excitation signals
are not informative as input feature. Therefore, in the acoustic
features of EL speech and silent EL speech, only the spectrum
is informative, and then, we use the spectra of EL speech andsilent EL speech as input feature to estimate the spectrum,
aperiodic components, andof normal speech uttered by non
laryngectomee.
In AL-to-Speech for these three types of alaryngeal speech,
the spectrum of each alaryngeal speech is used as input feature to estimate each acoustic feature of each type of alaryngeal speech, respectively. However, directly using the spectrumof alaryngeal speech causes the degradation of the converted
speech because the spectrum structures of some phonemes ofalaryngeal speech are often collapsed owing to difficulties of
producing them. To address these issues, we use a spectral segment feature extracted from multiple frames [25] as follows:
(10)
where
is a joint vector generated by concatenating a spectral parameter vector at the current frame and those at
preceding and succeeding frames.
Because this joint vector includes significantly redundant information, dimensionality reduction with principal component
analysis (PCA) is performed for the joint vector
in order toextract a spectral segment feature
at frame , where andDOI et al.: ALARYNGEAL SPEECH ENHANCEMENT BASED ON ONE-TO-MANY EVC

Fig. 3. Training process and conversion process of AL-to-Speech.

177

Fig. 4. Training process and adaptation process in AL-to-Speech based on” and “” show spectral features and aperiodic
one-to-many EVC. “, respectively.
components of normal speech of the th speaker

C. AL-to-Speech Based on EVC
are a transformation matrix and a bias vector extracted by PCA,
respectively.
B. AL-to-Speech Based on VC
AL-to-Speech based on VC consists of a training process anda conversion process shown in Fig. 3.
Training Process: In the training process, we record utterance pairs of alaryngeal speech and normal speech uttered bya non laryngectomee. To convert the spectral segment featureof alaryngeal speech into three speech parameters of the target
normal speech: namely, (1) spectral features, (2) log-scaled ,and (3) aperiodic components, we independently train three
GMMs modeling joint probability densities of the spectral
segment feature of alaryngeal speech and individual target
speech parameters using the corresponding joint feature vector
sets. For thefeature, a constant value clearly different fromvalues (e.g., a value much less than the minimum
value)
is used to represent unvoiced frames [26]. In
estimation for
ES speech,
extracted from normal speech recorded so as tobe similar to the pitch of ES speech is used as the target.
Conversion Process: In the conversion process, spectral segment features are extracted from alaryngeal speech. Then, individual converted speech parameters are independently estimated from the extracted spectral segment features using eachof the trained GMMs. In theestimation, unvoiced/voiced decision is also performed using a manually setting threshold todetect the constant value to represent unvoiced frames. After
estimating the converted spectral features, the converted logscaled , and the converted aperiodic components, the excitation signal is generated using STRAIGHT mixed excitation on
the basis of the converted
values and the converted aperiodic
components [15]. Finally, the converted speech is synthesizedby filtering the generated excitation signal with the converted
spectral features.

In AL-to-Speech based on the basic VC, it is difficult torecover speaker individuality of alaryngeal speech owing to
the predefined voice quality of converted speech, although thesound quality of converted speech is improved. To flexibly control the converted voice quality, we further apply one-to-many
EVC to AL-to-Speech. The EVC technique allows users tomanually control the voice quality of converted speech. Furthermore, conversion models can be adapted using a small
number of utterances of target speech in a text-independent
manner. Therefore, if recorded normal speech data of thelaryngectomee before undergoing the total laryngectomy still
exist, AL-to-Speech based on EVC can estimate the converted
speech that sounds similar to the laryngectomee’s original
voice. In this section, we describe the AL-to-Speech systemon the basis of the assumption that a few utterances of thelaryngectomee’s original speech have been kept. AL-to-Speechbased on EVC consists of training, adaptation, and conversion
processes. Fig. 4 shows the training and adaptation process in
AL-to-Speech based on one-to-many EVC.
Training Process: In the training process, we independently
train two one-to-many EV-GMMs: namely, a one-to-many
EV-GMM for estimating the converted spectral feature and a
one-to-many EV-GMM for estimating the converted aperiodic
components, for each alaryngeal speech. To train these two
EV-GMMs, we use multiple parallel data sets consisting of alaryngeal speech data uttered by the laryngectomee and prestored
normal speech data uttered by many non laryngectomees. The
EV-GMM for the spectral estimation is trained using multiple
joint feature vector sets consisting of the spectral segment features of the laryngectomee and the prestored spectral features of
the non laryngectomees. On the other hand, the EV-GMM forthe aperiodic component estimation is trained using multiple
joint feature vector sets consisting of the spectral segment
features of the laryngectomee and the prestored aperiodic components of the non laryngectomees. In this paper, we perform178

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 22, NO. 1, JANUARY 2014

the training method on the basis of Speaker Adaptive Training(SAT) [27] for the EV-GMM [28].
The EV-GMM is adapted to a new target speaker by adjusting the weight vector so that the marginal likelihood forgiven target speech features is maximized as shown in (9). This
adaptation process is effective if speaker-dependent characteristics are well captured by short-term features, such as spectrum
and aperiodic components. On the other hand, it is essentiallydifficult to control speaker-dependent characteristics captured
by long-term features, such as
patterns. Therefore, instead of
the EV-GMM, a well-trained speaker-dependent GMM is used
to estimate thepatterns from the spectral segment sequenceof alaryngeal speech. In AL-to-Speech for ES speech, to develop the GMM for estimating thepatterns corresponding to
the perceived pitch information of ES speech, we use
valuesextracted from normal speech uttered by a non laryngectomee
as an imitating prosody of ES speech in the training as the output
features. This process is the same as in the AL-to-Speech basedon VC described in Section IV-B. To develop a GMM for
estimation in EL speech and silent EL speech, speaker-dependent GMMs are separately trained for all target speakers. Then,the GMM achieving the highest
estimation accuracy is manually selected.
Adaptation and Conversion Processes: Assuming that a
few speech samples uttered by laryngectomees before undergoing total laryngectomy are available as adaptation data, theEV-GMM is flexibly adapted to the target voice quality byautomatically determining the weight vector in a text-independent manner [12]. The weight vectors of the EV-GMMs for the
spectral and aperiodic estimations are independently estimatedusing the spectral features and aperiodic components extracted
from the given target speech samples. The converted spectralfeature vectors and aperiodic components are independently
estimated using the adapted EV-GMMs. In theestimation,
the global speaker-dependent characteristics ofpatterns are
simply controlled. A log-scaled
sequence is first estimatedwith the selected speaker-dependent GMM, and then further
converted so that its meanand standard deviationareequal to those of the adaptation speech data,
and , asfollows:(11)
where
and denote thevalue estimated with the GMM
and the converted
value at frame , respectively.
V. EXPERIMENTAL EVALUATIONS
To demonstrate the effectiveness of the AL-to-Speech basedon VC/EVC methods, we conducted experimental evaluations
using several criteria. Then, we explicitly indicate the advantage
of each alaryngeal speech when applying AL-to-Speech.
A. Experimental Conditions
We recorded 50 phonetically balanced sentences of ES
speech uttered by one Japanese male laryngectomee, those of
EL speech and silent EL speech uttered by another Japanesemale laryngectomee, and those of normal speech uttered byeach of 40 Japanese non laryngectomees. The speech data of

30 non laryngectomees were used for training and those of theother 10 non laryngectomees were used as the target data forevaluation. From the 50 recorded sentences of each speaker, 40
were used as the training or adaptation data and the remaining
10 were used as the test data. The sampling frequency was set
to 16 kHz.
The 0th through 24th mel-cepstral coefficients were used as
spectral parameters. STRAIGHT analysis [17], which isadaptive analysis to extract accurate spectral envelope by effectively removing the effect of
periodicity on spectrum, wasemployed for normal speech. On the other hand, mel-cepstrum
analysis [29] was employed for alaryngeal speech sinceofalaryngeal speech is not informative.
As the source excitation features of normal speech, weused log-scaled
values and aperiodic components on fivefrequency bands, i.e., 0-1, 1-2, 2-4, 4-6, and 6-8 kHz, whichwere used for designing mixed excitation. The frame shift was
5 ms. To extract the spectral segment feature of ES speech,
current andframes were used for spectral and aperiodic
component estimations and current andframes were usedforestimation. For EL speech and silent EL speech, current
andframes were used for each parameter estimation. These
numbers of frames per segment were experimentally optimized[13].
The EV-GMMs for spectral and aperiodic component estimations were trained for each type of alaryngeal speech. The
numbers of eigenvectors and mixture components were set to
29 and 64 in every EV-GMM, respectively. The EV-GMMs
were adapted to the target speakers using 1, 2, 4, 8, 16, or
32 utterances of their normal speech data. For AL-to-Speechbased on VC, the GMMs for spectral and aperiodic estimation
were trained using a parallel dataset for each type of alaryngeal
speech and normal speech of each target speaker. The numberof training utterance pairs was set to 1, 2, 4, 8, 16 or 32.
The number of mixture components was optimized manuallydepending on the training data size so that the best conversion accuracy in the evaluation data was obtained. Individual
speaker-dependent GMMs for
estimation were trained for
all the 40 non laryngectomees. The GMM yielding the mostnaturalpattern was then selected by listening to the converted speech. The same
estimation process was performed
for the EVC-based AL-to-Speech and VC-based AL-to-Speech.
Full covariance matrices were used in every GMM/EV-GMM.
B. Objective Evaluations
We evaluated the effectiveness of AL-to-Speech for each typeof alaryngeal speech with estimation accuracy of each acoustic
feature, i.e., spectrum, aperiodic components, and .
Estimation Accuracy of Spectrum and Aperiodic Components: Figs. 5 and 6 show mel-cepstral distortion and root
mean square error (RMSE) on aperiodic components as afunction of the number of adaptation utterances used in EVC
or of utterance pairs used in VC, respectively. EVC shows asignificantly smaller mel-cepstral distortion and RMSE than
VC in each type of alaryngeal speech enhancement when theamount of the target normal speech data is small. Even if only
one arbitrary utterance of the target normal speech is available
in EVC, its conversion performance is almost equivalent to or

DOI et al.: ALARYNGEAL SPEECH ENHANCEMENT BASED ON ONE-TO-MANY EVC

179

TABLE IIESTIMATION ACCURACIES FOR VARIOUS TARGET SPEAKERS USINGCORRESPONDING TARGET-SPEAKER-DEPENDENT GMMS

TABLE IIIESTIMATION ACCURACIES FOR ACTUAL TARGET SPEAKERS IN EVALUATIONUSING WELL-TRAINED SPEAKER-DEPENDENT GMMS

Fig. 5. Mel-cepstral distortion as a function of the number of utterances oftarget normal speech (i.e., utterance pairs in VC or adaptation utterances in
EVC).

speech and silent EL speech, the estimation accuracy of the selected GMMs is higher than that of various speaker-dependent
GMMs shown in Table II, even though a speaker different fromthe target speakers is used in the training. To generate a natural
pattern in AL-to-Speech, it is useful to select an optimum
speaker for training rather than to directly use the same speakeras the actual target speaker since theestimation accuracy
largely varies among different speakers. It is also observed thatES speech enhancement yields better
correlation than theothers.
C. Subjective Evaluations
Fig. 6. RMSE on aperiodic components as a function of the number of utterances of target normal speech (i.e., utterance pairs in VC or adaptation utterances in EVC).

better than that of VC using 16 parallel utterance pairs. It is also
observed that ES speech yields the best conversion accuracy,
and silent EL speech yields the worst among the three types ofalaryngeal speech.
Estimation Accuracy: We also evaluated theestimation accuracy in AL-to-Speech for each type of alaryngeal
speech using
correlation coefficient and Unvoiced/Voiced(U/V) error between converted speech and target normal
speech. To demonstrate theestimation accuracy for various speakers in AL-to-Speech, the results calculated using
individual speaker-dependent GMMs for the 40 non laryngectomees are shown in Table II. For ES speech, the results foranother non laryngectomee who uttered normal speech so thatits pitch sounded similar to that of ES speech are also shownas “ES pitch.” ES speech yields the best estimation accuracy
among the three types of alaryngeal speech. Additionally, theestimation accuracy is significantly improved using the GMM
developed with the normal speech, thepatterns of whichcorrespond well to the pitch patterns of ES speech.
The final results for the 10 target non laryngectomees fromthe test data are shown in Table III. The GMM for “ES pitch”
was used in ES speech enhancement, and manually selected
speaker-dependent GMMs were used in the EL/silent EL speechenhancement. Namely, the speaker used in the model training isdifferent from the target speakers. It is observed that, for EL

We evaluated the effectiveness of AL-to-Speech for eachtypes of alaryngeal speech with speech quality, listenability,
intelligibility, and speaker individuality. In this paper, the term
“listenability” is used to indicate a score that was measured
by asking the listener to subjectively evaluate how easy it
was to understand the utterance. On the other hand, the term
“intelligibility” is used to indicate a score that was calculated
by asking the listener to write down the content of the utterance,and measuring the accuracy of transcription.2
Opinion Test on Speech Quality and Listenability: We conducted opinion tests on speech quality and listenability. In theopinion test of speech quality and listenability, 8 listeners evaluated 9 types of speech including original alaryngeal speechand converted speech with AL-to-Speech based on VC/EVC
in ES speech, EL speech, and silent EL speech. The VC-based
AL-to-Speech used 32 utterance pairs for GMM training. On
the other hand, only one utterance was used as adaptation data
for the EVC-based AL-to-Speech. The GV was considered in
the conversion process. For the EVC-based AL-to-Speech, themean vector of the GV probability density was set to the GV extracted from the adaptation utterance for each target non laryngectomee and the covariance matrix was fixed to that calculatedusing the GVs extracted from all utterances of the non laryngectomees used in the training of the EV-GMM. The opinion
score in each test was set to a 5-point scale. Individual speechsamples were normalized so that loudness of each sample wasalmost the same as each other. Note that signal-to-noise ratio
was kept in each sample before and after the normalization. We
2Note that the term “intelligibility” in [9] is used in the sense of “listenability.”180

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 22, NO. 1, JANUARY 2014

TABLE IVRESULT OF DICTATION TEST ON INTELLIGIBILITY

Fig. 7. Result of opinion test of speech quality. “Org”, “VC”, and “EVC” indicate original alaryngeal speech, converted speech by AL-to-Speech based on
VC trained with 32 utterance pairs, and converted speech by AL-to-Speechbased on EVC adapted with one utterance of target speech, respectively.

Fig. 9. Result of preference test of speaker individuality.

Fig. 8. Result of opinion test of listenability.

asked the listeners to evaluate speech samples with a wide rangeof the score from 1 to 5; i.e., a higher score was better speech
quality or listenability. In each test, the individual listeners listened to several speech samples before starting the test to maketheir own score range as stable as possible. Each listener evaluated 135 speech samples in the test.
Figs. 7 and 8 show the results of the opinion tests of speech
quality and listenability, respectively. All the AL-to-Speechmethods yield significant improvements in speech quality compared with that of the original alaryngeal speech. The speech
quality of the enhanced silent EL speech is lower than thoseof the enhanced ES speech and enhanced EL speech but it issignificantly higher than that of each type of original alaryngeal
speech. The listenability of ES speech and silent EL speech
are also improved by AL-to-Speech. On the other hand, thelistenability of EL speech slightly degrades from that of theoriginal EL speech by AL-to-Speech, as observed in our previous work [3]. The speech quality and listenability enhancedby the EVC-based AL-to-Speech are almost equivalent tothose enhanced by the VC-based AL-to-Speech. Note that the
EVC-based method requires only one arbitrary utterance of thetarget normal speech, whereas the VC-based method requires
32 utterance pairs of alaryngeal speech and the target normal
speech.

Dictation Test on Intelligibility: We conducted a manual dictation test. In this test, 6 listeners evaluated 6 types of speech
including original alaryngeal speech and converted speech by
AL-to-Speech based on EVC. Only one utterance was used as
adaptation data in the adaptation process of EV-GMMs. We allowed listeners to replay the same stimulus as much as theywant.
Table IV shows word correct, word accuracy, and the average
number of replays by listeners. Before conversion, EL speechis the best, ES speech is the next, and silent EL speech is theworst in intelligibility. We found that articulation of the laryngectomee who uttered EL speech and silent EL speech is clearer
than that of the other laryngectomee who uttered ES speech. Because quality of silent EL speech is significantly degraded by thesmall-powered excitation and body-conductive recording, silent
EL speech is less intelligible than ES speech even if it is moreclearly articulated than ES speech.
Intelligibility of silent EL speech is significantly improved by
AL-to-Speech based on EVC. On the other hand, intelligibilityof ES speech and EL speech are degraded by AL-to-Speechbased on EVC. Consequently, after conversion, EL speech isstill the best, silent EL speech is the next, and ES speech is theworst. In AL-to-Speech, conversion errors are inevitable andthey tend to cause degradation in intelligibility. Moreover, if articulation of input speech is unclear and unstable, larger degradation in intelligibility tends to be caused by conversion as observed in ES speech.
On the other hand, intelligibility of silent EL speech is significantly improved by conversion. As articulation of silent EL
speech is relatively clear and stable, the degradation of intelligibility caused by conversion tends to be smaller. Moreover,
AL-to-Speech well addresses a problem of very low quality

DOI et al.: ALARYNGEAL SPEECH ENHANCEMENT BASED ON ONE-TO-MANY EVC

181

contours, and aperiodic components of (a) normal speech and three types of converted
Fig. 10. Example of acoustic features, i.e., waveforms, spectrograms,
speech by AL-to-Speech based on EVC from (b) ES speech, (c) EL speech, and (d) silent EL speech in the same sentence fragment /h o N sy o h a k o t o b an o/. In aperiodic components, the solid line, coarse broken line, and fine broken line represent low band, middle band, and high band of aperiodic components,
respectively.

causing degradation in intelligibility of silent EL speech. Consequently, intelligibility of silent EL speech is better than ES
speech after conversion. These results suggest that our proposed
method is basically capable of improving intelligibility of alaryngeal speech if its intelligibility is degraded by some factors,
such as low speech quality, except for less articulated sounds.
Preference Test on Speaker Individuality: We also conducted
ABX test as a preference test to evaluate speaker individuality.
In the preference test, 6 listeners evaluated 6 types of speech
consisting of converted speech by the VC/EVC-based AL-toSpeech in EL speech, ES speech, and silent EL speech. In thistest, listeners heard target speech sample and two speech samples from among 6 types of converted speech, and then, theychose speech sample that has more similar speaker individuality
as target speech sample.
Each listener evaluated 60 pairs in the test.
The training data used in VC and the adaptation data used in
EVC were the same as those used in the opinion tests.
Fig. 9 shows the result of the preference test.
Every type of the converted speech was compared against
all the others and the preference score of each was calculated
as the ratio of the number of samples selected as having betterspeaker individuality to the total number of samples presented
to listeners.
We can observe the same tendency as that in Fig. 6. Enhanced
ES speech yields the best speaker individuality and enhanced
silent EL speech yields the worst among the three types of alaryngeal speech. Even if using only one arbitrary utterance ofthe target speaker is used in the EVC-based method, its performance is close to that of the VC-based method using 32 parallel utterances of the target speaker. This result shows that the

EVC-based method is capable of effectively adjusting speakerindividuality of the converted speech.
D. Example of the Converted Speech by AL-to-Speech Basedon EVC
Fig. 10 shows an example of the acoustic features of target
normal speech and converted speech from the three types of alaryngeal speech by AL-to-Speech based on EVC. These samples
were converted from each alaryngeal speech shown in Fig. 2.
We can see that the acoustic features of each converted speechcome closer to those of normal speech than to those of eachtype of alaryngeal speech. In the spectrogram, the spectral structure of converted speech from ES speech became clearer andstably varies compared with those of ES speech. Moreover, thespectral structure at high frequency that could not be observedin EL speech and silent EL speech is observed in converted
speech from EL speech and silent EL speech.
of converted
speech from each type of alaryngeal speech can capture thecoarse structure of those of normal speech. Furthermore, although over smoothing occurs, the aperiodic components of
converted speech are similar to those of normal speech. Therefore, the AL-to-Speech based on EVC is significantly effective
for the enhancement of alaryngeal speech.
VI. CONCLUSIONSIn this paper, we presented AL-to-Speech as enhancement methods based on VC and one-to-many EVC for three
types of alaryngeal speech, namely, esophageal speech (ESspeech), electrolaryngeal speech (EL speech), and body-conducted silent electrolaryngeal speech (silent EL speech). Thesemethods convert a spectral segment feature into spectrum,182

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 22, NO. 1, JANUARY 2014

aperiodic components, andof normal speech independently
using different GMMs or EV-GMMs. The experimental results suggested that (1) the proposed methods significantly
improve the speech quality of each type of alaryngeal speech,(2) the proposed methods also improve the listenability of ES
speech and silent EL speech, (3) the proposed methods alsosignificantly improve the intelligibility of silent EL speech,(4) AL-to-Speech based on eigenvoice conversion (EVC) is
capable of effectively adjusting the voice quality of enhanced
speech to the target voice quality using only one arbitraryutterance of the target voice.
Although the individual types of alaryngeal speech has theirown merits and demerits in terms of speech quality, listenability,
intelligibility, speaker individuality, difficulties of learning how
to produce, and so on. Some of the demerits are effectively addressed by AL-to-Speech. The results presented in this paper
could be helpful for laryngectomees to decide which type ofalaryngeal speech they use. We plan to develop and evaluate the
AL-to-Speech systems for other alaryngeal speech and further
implement a real-time conversion process for AL-to-Speech.

ACKNOWLEDGMENT
The authors are grateful to Prof. Hideki Kawahara of
Wakayama University, Japan for permission to use the
STRAIGHT analysis-synthesis method.

REFERENCES[1] K. Nakamura, T. Toda, H. Saruwatari, and K. Shikano, “Evaluation of
extremely small sound source signals used in speaking-aid system withstatistical voice conversion,” IEICE Trans. Inf. Syst., vol. E93-D, no.
7, pp. 1909–1917, Jul. 2010.
[2] Y. Nakajima, H. Kashioka, N. Campbell, and K. Shikano, “Non-Audible Murmur (NAM) recognition,” IEICE Trans. Inf. Syst., vol.
E89-D, no. 1, pp. 1–8, Jan. 2006.
[3] K. Nakamura, T. Toda, H. Saruwatari, and K. Shikano, “The use of
air-pressure sensor in electrolaryngeal speech enhancement basedon statistical voice conversion,” in Proc. Interspeech, Sep. 2010, pp.
1628–1631.
[4] A. Hisada and H. Sawada, “Real-time clarification of esophageal
speech using a comb filter,” in Proc. ICDVRAT, Sep. 2002, pp. 39–46.
[5] K. Matsui, N. Hara, N. Kobayashi, and H. Hirose, “Enhancement ofesophageal speech using formant synthesis,” in Proc. ICASSP, May
1999, pp. 1831–1834.
[6] H. R. Sharifzadeh, I. V. McLoughlin, and F. Ahmadi, “Reconstruction of normal sounding speech for laryngectomy patients through a
modified CELP codec,” IEEE Trans. Biomed. Eng., vol. 57, no. 10, pp.
2448–2458, Oct. 2010.
[7] H. Liu, Q. Zhao, M. Wan, and S. Wang, “Enhancement of electrolarynx
speech based on auditory masking,” IEEE Trans. Biomed. Eng., vol.
53, no. 5, pp. 865–874, May 2006.
[8] G. Aguilar-Torres, M. Nakano-Miyatake, and H. Perez-Meana, “Enhancement and restoration of alaryngeal speech signals,” in Proc. 16th
IEEE Int. Conf. Electron., Commun. Comput. (CONIELECOMP ’06),
Feb. 2006, p. 30.
[9] H. Doi, K. Nakamura, T. Toda, H. Saruwatari, and K. Shikano, “Anevaluation of alaryngeal speech enhancement methods based on voice
conversion techniques,” in Proc. ICASSP, May 2011, pp. 5136–5139.
[10] M. Abe, S. Nakamura, K. Shikano, and H. Kuwabara, “Voice conversion through vector quantization,” J. Acoust. Soc. Jpn. (E), vol. 11, no.
2, pp. 71–76, 1990.
[11] Y. Stylianou, O. Cappe, and E. Moulines, “Continuous probabilistictransform for voice conversion,” IEEE Trans. Speech Audio Process.,
vol. 6, no. 2, pp. 131–142, Mar. 1998.

[12] T. Toda, Y. Ohtani, and K. Shikano, “One-to-many and many-to-one
voice conversion based on eigenvoices,” in Proc. ICASSP, Apr. 2007,
pp. 1249–1252.
[13] H. Doi, K. Nakamura, T. Toda, H. Saruwatari, and K. Shikano,“Esophageal speech enhancement based on statistical voice conversion with Gaussian mixture models,” IEICE Trans. Inf. Syst., vol.
E93-D, no. 9, pp. 2472–2482, Sep. 2010.
[14] H. Kawahara, H. Katayose, A. Cheveigne, and R. D. Patterson, “Fixed
point analysis of frequency to instantaneous frequency mapping forand periodicity,” in Proc. EUROSPEECH,
accurate estimation of
Sep. 1999, pp. 2781–2784.
[15] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Maximum likelihood voice conversion based on GMM with STRAIGHT mixed excitation,” Interspeech ’06–ICSLP, pp. 2266–2269, Sep. 2006.
[16] H. Kawahara, J. Estill, and O. Fujimura, “Aperiodicity extraction andcontrol using mixed mode excitation and group delay manipulation fora high quality speech analysis, modification and system straight,” inProc. MAVEBA, Sep. 2001.
[17] H. Kawahara, I. Masuda-Katsuse, and A. Cheveigne, “Restructuring speech representations using a pitch-adaptive time-frequency
extraction: Possmoothing and an instantaneous-frequency-based
sible role of a repetitive structure in sounds,” Speech Commun., vol.
27, no. 3-4, pp. 187–207, Apr. 1999.
[18] T. Hirahara, M. Otani, S. Shimizu, T. Toda, and K. Nakamura, “Silentspeech enhancement using body-conducted vocal-tract resonance signals,” Speech Commun., vol. 52, no. 4, pp. 301–313, Apr. 2010.
[19] T. Toda, A. W. Black, and K. Tokuda, “Voice conversion based on maximum likelihood estimation of spectral parameter trajectory,” IEEE
Trans. Audio, Speech, Lang. Process., vol. 15, no. 8, pp. 2222–2235,
Nov. 2007.
[20] A. Kain and M. W. Macon, “Spectral voice conversion for text-tospeech synthesis,” in Proc. ICASSP, May 1998, pp. 285–288.
[21] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura,“Speech parameter generation algorithms for HMM-based speech synthesis,” in Proc. ICASSP, Jun. 2000, pp. 1315–1318.
[22] R. Kuhn, J. Junqua, P. Nguyen, and N. Niedzielski, “Rapid speaker
adaptation in eigenvoice space,” IEEE Trans. Speech Audio Process.,
vol. 8, no. 6, pp. 695–707, Nov. 2000.
[23] B. Milner and X. Shao, “Prediction of fundamental frequency andvoicing from mel-frequency cepstral coefficients for unconstrained
speech reconstruction,” IEEE Trans. Speech Audio Process., vol. 15,
no. 1, pp. 24–33, Dec. 2007.
[24] J. Darch, B. Milner, and S. Vaseghi, “Analysis and prediction of
acoustic speech features from mel-frequency cepstral coefficients indistributed speech recognition architectures,” J. Acoust. Soc. Amer.,vol. 124, no. 6, pp. 3989–4000, Dec. 2008.
[25] T. Toda, A. W. Black, and K. Tokuda, “Statistical mapping between
articulatory movements and acoustic spectrum with a Gaussian mixture
model,” Speech Commun., vol. 50, no. 3, pp. 215–227, Mar. 2008.
[26] T. Toda, M. Nakagiri, and K. Shikano, “Statistical voice conversion
techniques for body-conducted unvoiced speech enhancement,” IEEE
Trans. Audio, Speech, Lang. Process., vol. 20, no. 9, pp. 2505–2517,
Sep. 2012.
[27] T. Anastasakos, J. McDonough, S. R. Schwartz, and J. Makhoul, “A
compact model for speaker-adaptive training,” in Proc. ICSLP, 1996,
vol. 2, pp. 1137–1140.
[28] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Adaptive training
for voice conversion based on eigenvoices,” IEICE Trans. Inf. Syst.,
vol. E93-D, no. 6, pp. 1589–1598, Jun. 2010.
[29] K. Tokuda, T. Kobayashi, T. Masuko, and S. Imai, “Mel-generalized
cepstral analysis–a unified approach to speech spectral estimation,” inProc. ICSLP, Sep. 1994, pp. 1043–1045.

Hironori Doi graduated from the Department ofInformation and Image Science, Faculty of Engineering, Chiba University, Japan in 2008. Hereceived his M.E. and D.E. degrees from the Graduate School of Information Science, Nara Instituteof Science and Technology (NAIST), Japan, in 2009
and 2012, respectively. He was a Research Fellowof JSPS in NAIST from 2011 to 2012. He currently
works for DWANGO Co., Ltd., Japan. He mainly
studies singing voice conversion and speaking-aid
systems.

DOI et al.: ALARYNGEAL SPEECH ENHANCEMENT BASED ON ONE-TO-MANY EVC

Tomoki Toda earned his B.E. degree from Nagoya
University, Aichi, Japan, in 1999 and his M.E.
and D.E. degrees from the Graduate School ofInformation Science, NAIST, Nara, Japan, in 2001
and 2003, respectively. He was a Research Fellowof JSPS in the Graduate School of Engineering,Nitech, Aichi, Japan, from 2003 to 2005. He wasan Assistant Professor of the Graduate School ofInformation Science, NAIST from 2005 to 2011,
where he is currently an Associate Professor. He has
also been a Visiting Researcher at the NICT, Kyoto,
Japan, since May 2006. From April 2003 to March 2006, he was a Visiting
Researcher at the ATR SLC Research Laboratories, Kyoto, Japan. He wasalso a Visiting Researcher at the LTI, CMU, Pittsburgh, USA, from October2003 to September 2004 and at the CUED, Cambridge, UK, from March toAugust 2008. His research interests include statistical approaches to speech
and language processing. He received 11 awards including the 2009 Young
Author Best Paper Award from the IEEE SPS and the 2013 EURASIP-ISCA
Best Paper Award (Speech Communication Journal). He was a member of the
SLTC of the IEEE SPS from 2007 to 2009.

Keigo Nakamura graduated from the Course ofMultimedia Studies, Faculty of Education and
Human Sciences, Yokohama National University,
Japan in 2005. He received his M.E. and D.E.
degrees from the Graduate School of Information
Science, Nara Institute of Science and Technology(NAIST), Japan, in 2007 and 2010, respectively.
He was a Research Fellow of JSPS in NAIST from2009 to 2011. He was also a Visiting Researcher at
the Karlsruhe Institute of Technology, Karlsruhe,
Germany, from April 2010 to February 2011. Hecurrently works for Rakuten, Inc., Japan. He mainly studies speaking-aid
systems for laryngectomees.

183

Hiroshi Saruwatari (M’00) was born in Nagoya,Japan, on July 27, 1967. He received the B.E., M.E.,
and Ph.D., degrees from Nagoya University in 1991,
1993, and 2000, respectively. He joined Intelligent
System Laboratory, SECOM Co., Ltd., Tokyo,
Japan, in 1993, where he engaged in the research on
the ultrasonic array system for the acoustic imaging.
He is currently an Associate Professor of Graduate
School of Information Science, Nara Institute of
Science and Technology. His research interests include noise reduction, array signal processing, blindsource separation, and sound field reproduction. He received paper awards fromIEICE in 2001 and 2006, from Telecommunications Advancement Foundationin 2004 and 2009, and from IEEE-IROS2005 in 2006. He won the first prizein IEEE MLSP2007 Data Analysis Competition for BSS. Prof. Saruwatari is amember of the IEICE, Japan VR Society, and the Acoustical Society of Japan.

Kiyohiro Shikano received the B.S., M.S., and Ph.D.
degrees in electrical engineering from Nagoya University in 1970, 1972, and 1980, respectively. He iscurrently a professor of Nara Institute of Science andTechnology (NAIST), where he is directing speech
and acoustics laboratory. From 1972, he had beenworking at NTT Laboratories, where he had been engaged in speech recognition research. During 19901993, he was the executive research scientist at NTT
Human Interface Laboratories, where he supervised
the research of speech recognition and speech coding.
During 1986-1990, he was the Head of Speech Processing Department at ATR
Interpreting Telephony Research Laboratories, where he was directing speech
recognition and speech synthesis research. During 1984-1986, he was a visiting scientist in Carnegie Mellon University. He received the IEICE (Instituteof Electronics, Information and Communication Engineers of Japan) Yonezawa
Prize in 1975, IEEE Signal Processing Society 1990 Senior Award in 1991, theTechnical Development Award from ASJ (Acoustical Society of Japan) in 1994,
IPSJ (Information Processing Society of Japan) Yamashita SIG Research Awardin 2000, Paper Award from the Virtual Reality Society of Japan in 2001, IEICE
Paper Award in 2005 and 2006, and IEICE Inose Best Paper Award in 2005. He
is a fellow member of IEEE, IEICE, and IPSJ. He is a member of ASJ, JapanVR Society, and International Speech Communication Association.