STATISTICAL APPROACH TO VOICE QUALITY CONTROLIN ESOPHAGEAL SPEECH ENHANCEMENT
Kenzo Yamamoto, Tomoki Toda, Hironori Doi, Hiroshi Saruwatari, Kiyohiro Shikano
Graduate school of Information Science, Nara Institute of Science and Technology, JAPAN
tomoki@is.naist.jp, hironori-d@is.naist.jp
ABSTRACTThis paper describes a voice quality control method in statisticalesophageal speech enhancement. Esophageal speech is produced
by one of the alternative speaking methods for laryngectomees. Its
naturalness and intelligibility are much lower than those of natural
voices and its voice quality sounds similar even if uttered by different laryngectomees. These issues are alleviated by a statistical voice
conversion method from esophageal speech into normal speech (ESto-Speech) based on eigenvoices. This method is capable of determining converted voice quality using a few target voice samples.
In this paper, we propose ES-to-Speech using regression techniques
to make it possible to manually control the converted voice quality
by manipulating a few intuitively controllable parameters even if notarget voice sample is available. The effectiveness of the proposed
method is conﬁrmed by experimental evaluations.
Index Terms— Esophageal speech, speech enhancement, voice
conversion, voice quality control, kernel regression
1. INTRODUCTION
Laryngectomees who have undergone a total laryngectomy due toan accident or laryngeal cancer cannot produce speech sounds in ausual way because their vocal folds have been removed. They needalternative speaking methods for producing speech sounds. The produced speech is called alaryngeal speech and esophageal speech isa type of alaryngeal speech. In producing esophageal speech, excitation signals are produced by releasing gases from or through theesophagus, and then they are articulated. Esophageal speech sounds
more natural than other types of alaryngeal speech, such as electrolaryngeal speech. However, the severe degradation of naturalness andintelligibility of esophageal speech caused by its speciﬁc production
mechanism is observed compared with normal speech. Moreover,its voice quality is similar even if different laryngectomees speak.
Consequently, esophageal speech also suffers from the severe degradation of speaker individuality.
To improve esophageal speech quality, several attempts basedon the modiﬁcations of acoustic features of esophageal speech using
signal processing, such as comb ﬁltering [1] or smoothing of acoustic parameters [2], have been carried out. Although they are useful inthe esophageal speech enhancement, quality improvements are still
limited since the acoustic features of esophageal speech exhibit quite
different properties from those of normal speech. Therefore, it is basically difﬁcult to compensate for those acoustic differences using
such a simple modiﬁcation process.
To signiﬁcantly improve the naturalness, intelligibility, andspeaker individuality of esophageal speech, a statistical method
for converting esophageal speech into normal speech (ES-toSpeech) has been proposed [3]. A statistical voice conversion (VC)
This research was supported in part by MIC SCOPE and MEXT Grantin-Aid for Young Scientists (A).

978-1-4673-0046-9/12/$26.00 ©2012 IEEE

4497

method [4, 5] is effectively used for converting acoustic featuresof esophageal speech to those of normal speech. Furthermore, torecover speaker individuality, one-to-many eigenvoice conversion(EVC) [6] has also been implemented for ES-to-Speech to makeit possible to ﬂexibly adapt voice quality of the converted speech
to that of given target speech samples. Although it is effective for
producing more varieties of voice quality, it does not work if any
target speech samples with desired voice quality are not available.
A technique for manually controlling voice quality is effective todevelop a more ﬂexible voice quality control framework.
As one of the statistical parametric speech synthesis techniques
capable of manually controlling voice quality of synthetic speech,
a multiple regression approach has been proposed in speech synthesis based on hidden Markov model (HMM) [7]. This regression
approach has also been applied to one-to-many EVC [8]. In thismethod, voice quality of various speakers is described by a few voice
quality control parameters based on primitive word pairs expressing speciﬁc voice quality factors [9]. To manually control converted
voice quality without any target voice samples, a subspace spannedby a few representative vectors capturing the speciﬁc voice quality
factors is formed in a statistical conversion model by extending aneigenvoice-based acoustic modeling technique [10].
Inspired by these methods, we propose voice quality control
methods in ES-to-Speech. Manual control of voice quality of theconverted speech is achieved by using a multiple regression Gaussian mixture model (MR-GMM) in ES-to-Speech. To further improve the performance of voice quality control in ES-to-Speech, we
propose a method for enhancing accuracy of the voice quality control
parameters and also propose a more accurate voice quality control
method based on kernel regression GMM (KR-GMM). The resultsof several experimental evaluations are reported to show the effectiveness of the proposed methods.
2. ES-TO-SPEECH BASED ON ONE-TO-MANY EVC
2.1. Training
Using multiple parallel datasets between esophageal speech of a laryngectomee and normal speech of many pre-stored target speakers,
the joint probability density function (p.d.f.) of a source feature vector of esophageal speech, X t , and a target feature vector of the sth(s)
pre-stored target speaker’s normal speech, Y t , at frame t is modeled by a one-to-many eigenvoice GMM (EV-GMM) [6] as follows:(s)
P X t , Y t |w(s) , λ  (X)   (XX)
M
(XY )

Xt
Σm
μm
Σm
=
αm N
, (1)
(s) ;
(Y,s) ,(Y X)
(Y Y )
Yt
μm
Σm
Σm
m=1
(Y )

) (s)
= B (Y+ bm,0 ,μ(Y,s)m w
m(2)

ICASSP 2012

where N (·; μ, Σ) denotes a Gaussian distribution with a mean vector μ and a covariance matrix Σ. The total number of mixturecomponents is M . The mixture-component weight αm , the source
(XX)
(XY )
mean vector μ(X), Σm ,
m , and the covariance matrices Σm(Y X)(Y Y )
Σm , and Σmare tied over every target speaker. On the other
hand, the target mean vector of the sth pre-stored target speakeris factorized into mixture-component-dependent parametersμ(Y,s)
m(Y )tied over every target speaker, i.e., a bias vector bm,0 and repre(Y )

(Y )

(Y )

sentative vectors B m = bm,1 , · · · , bm,J , and a target-speakerdependent parameter tied over every mixture-component, i.e., an
(s)

(s)



. Consequently, the EVadaptive vector w(s) = w1 , · · · , wJ
GMM has the adaptive vector and a ﬁxed parameter set λ consistingof the other parameters.
To convert esophageal speech into normal speech, some parameters of normal speech, such as spectrum, aperiodic components, andF0 , are estimated from a spectral parameter of esophageal speech[3]. For the estimation of spectrum and aperiodic components, we
independently train two EV-GMMs modeling the joint p.d.f.s of thespectral segment feature of esophageal speech and two features of
the normal speech parameters, spectrum and aperiodic components,using corresponding joint feature vector sets. On the other hand, as
for the F0 estimation, a naturally varying F0 pattern corresponding
to perceivable pitch information of esophageal speech is estimated.
In other words, a statistical F0 estimation process for esophageal
speech is performed using the VC technique. To achieve suchan estimation process, normal speech carefully uttered by a nonlaryngectomee so that its pitch sounds similar to that of esophageal
speech is newly recorded. Then, its extracted F0 is used as the target
speech parameter to train a standard GMM modeling the joint p.d.f.
of the spectral segment feature of esophageal speech and the target
F0 feature corresponding to pitch of esophageal speech.
2.2. Adaptation and Conversion
As for the adaptation of spectrum and aperiodic components, each
EV-GMM is separately adapted to given target speech samples in an
unsupervised manner. An optimum value of the adaptive vector w
is determined by maximizing a marginal likelihood P (Y |w, λ) of
the EV-GMM for the given target speech features Y . For the F0
adaptation, global mean and standard deviation values are extracted
from the given target speech samples.
In conversion, spectrum and aperiodic components are separately estimated from the spectral segment feature of esophageal
speech using the corresponding adapted EV-GMMs. F0 is estimatedfrom the spectral segment feature using the standard GMM. The
maximum likelihood estimation method considering dynamic features and the global variance [5] is used in these estimation processes. To adapt global F0 characteristics to those of the given target
speech samples, the estimated F0 pattern is linearly transformed so
as to its mean and standard deviation values over an utterance isequivalent to the target values.
2.3. Limitation
In ES-to-Speech based on one-to-many EVC, only a few arbitraryutterances of the target speech can be used as adaptation data. For
instance, even if there were only a small amount of the recorded
original voices of laryngectomees before a total laryngectomy, their
original voice quality would be recovered. However, those original
voices are not always available. Moreover, other voices different
from their original ones would be preferred by some people. Therefore, the development of a technique for allowing laryngectomees tocustomize voice quality as they want is desired.

4498

3. VOICE QUALITY CONTROL IN ES-TO-SPEECH
We propose voice quality control methods in ES-to-Speech to makeit possible to manually control converted voice quality. It is essential
in voice quality control to design an intuitively controllable parameter to be manipulated. One promising approach is to use perceptual
scores expressing speciﬁc voice quality factors. In the literature [9],several primitive word pairs to efﬁciently represent voice quality ofvarious speakers, such as male/female for gender or elder/younger
for age, have been extracted through a large-sized perceptual evaluation using normal speech of a lot of speakers. Based on this conventional work, we use perceptual scores on these primitive word pairs
as the voice quality control parameter.
3.1. Voice Quality Control Based on MR-GMM in ES-to-Speech
The use of the MR-GMM allows us to manually control converted
voice quality [8]. In training, the multiple parallel datasets including esophageal speech and normal speech of many pre-stored target
speakers are also used as in the conventional ES-to-Speech based on
one-to-many EVC. The perceptual scores on the primitive word pairs
are manually assigned to each pre-stored target speaker through listening to his/her natural voices. They are used to form a voice quality
control vector of each pre-stored target speaker: e.g., that of the sth
(s)

(s)

(s)



pre-stored target speaker is given by wc = wc,1 , · · · , wc,J ,where individual dimensional components (hereafter voice quality
control scores) are given by the manually assigned perceptual scoreson J primitive word pairs. Using the multiple parallel datasets andthe corresponding voice quality control vectors, the MR-GMM istrained. A model structure of the MR-GMM is the same as that of(s)
the EV-GMM but the voice quality control vector wc is used in(s)stead of the adaptive vector w . Since the voice quality control
vectors of individual pre-stored target speakers are ﬁxed during thetraining, the resulting model parameters of the MR-GMM are different from those of the EV-GMM. While the representative vectors of
the EV-GMM capture dominant voice characteristics over the prestored target speakers, those of the MR-GMM capture speciﬁc voice
quality factors expressed by the primitive word pairs. Moreover, thedimensionality of the voice quality control vector is usually much
lower than that of the adaptive vector since the use of a small number of control parameters is preferable in terms of controllability.
Two MR-GMMs for converting the spectral segment feature ofesophageal speech into spectrum or aperiodic components of normal speech are independently trained. As for the F0 control, global
mean and standard deviation values of the F0 pattern are extracted
for each pre-stored target speaker, and then, a relationship between
those values and the voice quality control vectors over all pre-stored
target speakers is modeled by multiple regression analysis.
In the conversion process, the voice quality control vector is
manually determined by manipulating each voice quality control
score to express the desired voice quality. Then, spectrum andaperiodic components exhibiting the desired voice quality are independently estimated from the spectral segment feature of esophageal
speech using the individual MR-GMMs adapted with the determinedvoice quality control vector. The F0 pattern is estimated in the samemanner as in the conventional ES-to-Speech, and then, it is globally
converted so that its mean and standard deviation values are equal tothose values estimated from the voice quality control vector.
3.2. Assignment of Voice Quality Control Scores
In the traditional MR-GMM training [8], natural voices of each prestored target speaker are used in the assignment of the voice qual-

ity control scores. Therefore, various acoustic features of natural
speech, such as local F0 patterns, duration, and so on, affect the resulting scores more or less even if they are not well controlled in
ES-to-Speech. Even in the spectral conversion, some spectral structures affecting voice quality perception would be lost by the effectof statistical generalization. The mismatch between acoustic features
affecting the score assignment and those actually controlled in ESto-Speech causes the performance degradation in voice quality control. To minimize this mismatch, we propose the use of the converted
speech in the score assignment. Esophageal speech is converted
into each pre-stored target speaker’s voice with the target-speakerdependent GMMs in the conventional ES-to-Speech and the scoreassignment is performed through listening to the converted speechrather than natural voices. Since the converted speech does not include any varieties of the acoustic features not well modeled by theGMMs, the resulting scores more precisely capture only varietiesof the voice quality practically controlled in ES-to-Speech than thescores assigned by listening to natural voices.
3.3. Voice Quality Control Based on Kernel Regression GMM(KR-GMM) in ES-to-Speech
In the MR-GMM, a relationship between each voice quality control
vector and the target mean vectors is assumed to be linearly modeled. If such an assumption does not hold, the performance of voice
quality control is degraded. To model a more complicated relationship between them, we propose a voice quality control method basedon the KR-GMM.
The voice quality control vector is mapped into a high dimensional feature space and linear regression is performed there. In the(s)
KR-GMM, given the voice quality control vector wc of the sththpre-stored target speaker, the d dimensional component of the tar(Y,s)(Y,s)μm,1 , · · · , μm,Dget mean vector μ(Y,s)
=
m
(Y,s)

=

μm,d

J




(3)

j=1

where φ(·) denotes the function to map each voice quality control
score to the feature space. A vector in the feature space v j,m,d canbe represented by
v j,m,d

S


=



(s)
αj,m,d,s φ wc,j ,

(4)

s=1

where the number of the pre-stored target speakers is S and theweighting parameter for each data sample is αj,m,d,s , which is optimized in kernel regression. Using Eq. (4), Eq. (3) is written as


(Y,s)
μm,d
= αm,d k w(s)
,
(5)
c
where
αm,d
αj,m,d==k(wc )=

kj (wc,j )=[α1,m,d , · · · , αJ,m,d ] ,[αj,m,d,1 , ..., αj,m,d,S ] ,

(6)
(7)



k
,
(8)
1 (wc,1 ) , · · · , kJ (wc,J )



 
(1)
(S)
k wc,j , wc,j , · · · , k wc,j , wc,j, (9)and k(·, ·) denotes the kernel function. In this paper, the radial basis
function (RBF) kernel given below is used:k (x, x )=exp (−β||x − x ||) ,+γαm,d α
(11)
m,d ,




(1)
(S)
(1:S)
where K, · · · , k wc
and a parameter for= k wcL2 norm regularization is γ. The weighting parameters optimizedby the minimization of the objective function are given by
−1



(Y,1:S).
(12)
K (1:S) K (1:S) + γI
α̂m,d = μ̂m,d K (1:S)
Consequently, the target mean vectors of the KR-GMM are given byμ(Y,s)= α̂k w(s).
(13)m,1 , · · · , α̂m,Dcm
Two KR-GMMs for the conversion into spectrum and into aperiodic components are separately trained. The kernel regression is also
used for the estimation of the F0 mean and standard deviations forgiven the voice quality control vector. In this paper, the parameter ofkernel function β and the regularization parameter γ are optimized
with cross-validation to maximize the conversion performance.
4. EXPERIMENTAL EVALUATIONS
4.1. Experimental Conditions

is modeled by



(s)
v
j,m,d φ wc,j ,As the dependent variables in kernel regression, the target mean
vectors of the individual target-speaker-dependent GMMs are used,which are developed by updating only the target mean vectors of the
MR-GMM using a parralel dataset corresponding to each pre-stored
target speaker. Note that they are no longer represented on the subspace of the MR-GMM since their factorized form is ignored in the
update. Let the dth dimensional components of the updated target
mean vectors of the mth mixture-component over all pre-stored tar(Y,1:S)
(Y,1)
(Y,S)
get speakers be μ̂m,d= μ̂m,d , · · · , μ̂m,d . The objectivefunction of kernel regression is given by(Y,1:S)(Y,1:S)
2m,d = μ̂m,d − αm,d K (1:S) μ̂m,d − αm,d K (1:S)(10)

where β is a parameter of the RBF kernel.

4499

We recorded 50 phoneme-balanced sentences of esophageal speechuttered by one Japanese male laryngectomee. The same sentencesof normal speech uttered by 61 Japanese non-laryngectomees (male34, female 27) were recorded for training the MR-GMMs/KRGMMs. Those of pitch-controlled normal speech uttered by one
Japanese non-laryngectomee were also recorded for training thestandard GMM used for the F0 estimation. Forty sentences out of
the recorded 50 sentences were used for training and the remaining
10 sentences were used for evaluation. Several parameters such as
the number of mixture-components of the MR-GMMs/KR-GMMs
were experimentally optimized.
The 0th through 24th mel-cepstral coefﬁcients were used as the
spectral parameter. As the excitation parameters of normal speech,
we used log-scaled F0 and aperiodic components on ﬁve frequency
bands (0-1, 1-2, 2-4, 4-6, and 6-8 kHz) for designing mixed excitation. STRAIGHT [11] was used in analysis of normal speech.
To deﬁne the voice quality control vector, we used a 5-scaled
score (-2: very, -1: somewhat, 0: no preference, 1: somewhat, 2:very) for 5 Japanese primitive word pairs expressing voice quality
factors, such as male/female (gender), husky/clear (clearness), elder/younger (age), deep/thin (deepness), and weak/strong (forcefulness). One Japanese male subject assigned these scores to each of the
pre-stored target speakers. The assigned scores for each word pair
were normalized into Z-score (zero mean and unit variance) over all
pre-stored target speakers. Although the results can not be shown
here due to lack of space, we conﬁrmed that the use of the scores assigned to the converted speech as described in Section 3.2 makes results of voice quality control more stable compared with those whenusing the scores assigned to natural voices.

thin

Perceptual score on deepness

We conducted subjective evaluations of voice quality control. The
number of listeners was 10. Five test sentences were used. For eachsentence, we synthesized 5 samples of the converted speech by varying only one voice quality control score from -2 to 2 in 5 steps whilesetting the other voice quality control scores to zero. The converted
speech samples when setting every voice quality control score to
zero were also synthesized as reference speech. Each listener compared the voice quality of the converted speech with that of the reference speech using a 5-scaled score (-2: very, -1: somewhat, 0: nodifference, 1: somewhat, 2: very) for the primitive word pair corresponding to that on the varied voice quality control score.
Figure 1 shows a result of voice quality control on deepness.
The perceptual score well correlates to the setting of the voice quality control score in both the EV-GMM and the KR-GMM. We havefound that good correlation between those two scores can also be observed when manipulating other voice quality control scores. These
results show that the proposed methods are capable of effectively
controlling voice quality of the converted speech in ES-to-Speechby manipulating the voice quality control vector.
We can also see that the KR-GMM is capable of generating converted speech of which voice quality varies more widely and is closer
to the setting of voice quality control score compared with the MRGMM. Discussion on a comparison between the MR-GMM and the
KR-GMM is shown below.

MR-GMM
KR-GMM
95% confidence intervals

2
1
0
-1
-2

deep

4.2.1. Evaluation of Controllability

-2

0

1

2

5
MR-GMM
KR-GMM
95% confidence intervals

4
3
2
1
-5

-4

deep
4.2.2. Evaluation of Naturalness-1

deep Voice quality control score on deepness thin
Fig. 1. Result of evaluation of voice quality controllability.
Mean opinion score (MOS)

4.2. Experimental Results

-3

-2

-1

0

1

2

3

Voice quality control score on deepness

4

5

thin

Fig. 2. Result of evaluation of naturalness.

We also conducted opinion tests of naturalness. Naturalness of theconverted speech was evaluated using a 5-scaled opinion score (from1: very bad to 5: excellent). The other experimental conditions were
the same as those in the previous evaluations on controllability.
Figure 2 shows a result when manipulating only the voice quality control score on deepness. We can see that naturalness of theconverted speech starts to be degraded when the voice quality control score is set to too large or too small values. Therefore, it is betterto keep the voice quality control score in a reasonable score range.
Although there is no signiﬁcant difference between the MR-GMM
and the KR-GMM in this ﬁgure, the KR-GMM is capable of controlling target voice quality more widely than the MR-GMM using
the same range of voice quality control score as shown in Figure 1.
We have also found that the KR-GMM can signiﬁcantly reduce thedegradation of naturalness caused by setting the voice quality control score to too small values compared with the MR-GMM whenmanipulating another voice quality control score. These results suggest that the KR-GMM is more effective than the MR-GMM forvoice quality control in ES-to-Speech.
5. CONCLUSION
This paper has described novel methods to intuitively control voice
quality in esophageal speech enhancement based on statistical voice
conversion. The multiple regression Gaussian mixture model (MRGMM) has been implemented for a statistical conversion processfrom the esophageal speech into normal speech (ES-to-Speech).
Moreover, the kernel regression GMM (KR-GMM) has been proposed to further improve the controllability of the voice quality. The
experimental results have showed that 1) our proposed methods allow a laryngectomee to control voice quality of the converted speechby manipulating a few control parameters on primitive word pairs
expressing speciﬁc voice quality factors and 2) the KR-GMM yields
better performance in voice quality control than the MR-GMM.

4500

6. REFERENCES[1] A. Hisada and H. Sawada. Real-time clariﬁcation of esophageal speechusing a comb ﬁlter. Proc. 4th Intl Conf. Disability, Virtual Reality &Assoc. Tech., pp. 39–46, Veszprem, Hungary, 2002.
[2] K. Matsui, N. Hara, N. Kobayashi, and H. Hirose. Enhancement ofesophageal speech using formant synthesis. Acoust. Sci. & Tech., Vol.
23, No. 2, pp. 69–76, 2002.
[3] H. Doi, K. Nakamura, T. Toda, H. Saruwatari, and K. Shikano.
Esophageal speech enhancement based on statistical voice conversion
with Gaussian mixture models. IEICE Trans. Inf. & Syst., Vol. E93-D,
No. 9, pp. 2472–2482, 2010.
[4] Y. Stylianou, O. Cappe, and E. Moulines. Continuous probabilistictransform for voice conversion. IEEE Trans. SAP, Vol. 6, No. 2, pp.
131–142, 1998.
[5] T. Toda, A.W. Black, and K. Tokuda. Voice conversion based on maximum likelihood estimation of spectral parameter trajectory. IEEE
Trans. ASLP, Vol. 15, No. 8, pp. 2222–2235, 2007.
[6] T. Toda, Y. Ohtani, and K. Shikano. One-to-many and many-to-one
voice conversion based on eigenvoices. Proc. ICASSP, pp. 1249–1252,
Hawaii, USA, Apr. 2007.
[7] T. Nose, J. Yamagishi, T. Masuko, and T. Kobayashi. A style control
technique for HMM-based expressive speech synthesis. IEICE Trans.
Inf. & Syst., Vol. E90-D, No. 9, pp. 1406–1413, 2007.
[8] K. Ohta, Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano. Regression
approaches to voice quality control based on one-to-many eigenvoice
conversion. 6th ISCA Speech Synthesis Workshop (SSW6), pp. 101–
106, Bonn, Germany, Aug. 2007.
[9] H. Kido and H. Kasuya. Everyday expressions associated with voice
quality of normal utterance —Extraction by perceptual evaluation—. J.
Acoust. Soc. Jpn., Vol. 57, No. 5, pp. 337-344, 2001 [in Japanese].
[10] R. Kuhn, J. Junqua, P. Nguyen, and N. Niedzielski. Rapid speaker
adaptation in eigenvoice space. IEEE Trans. SAP, Vol. 8, No. 6, pp.
695–707, 2000.
[11] H. Kawahara, I. Masuda-Katsuse, and A.de Cheveigné. Restructuring
speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible roleof a repetitive structure in sounds. Speech Communication, Vol. 27, No.
3-4, pp. 187–207, 1999.