GlobalSIP 2014: Machine Learning Applications in Speech Processing

Augmented Speech Production based on
Real-Time Statistical Voice Conversion
Tomoki Toda
Graduate School of Information Science, Nara Institute of Science and Technology
Takayama 8916-5, Ikoma, Nara, 630-0192 JAPAN
Email: tomoki@is.naist.jp
Abstract—In human-to-human speech communication, various without any text transcriptions. This VC framework has beenbarriers are caused by some constraints, such as physical con- studied mainly in the context of speaker conversion, i.e., a constraints causing vocal disorders and environmental constraints version from a voice of a certain speaker (the source speaker)
making it hard to produce intelligible speech. These barriers
would be overcome if our speech production was augmented so to sound like that of another speaker (the target speaker),that we could produce speech sounds as we want beyond these since late 80s [1]. The most popular approach to this VC
constraints. Voice conversion (VC) is a technique for modifying framework is based on statistical processing, which is capablespeech acoustics, converting non-/para-linguistic information to of extracting a complex conversion function in a data-driven
any form we want while preserving the linguistic content. One manner. The statistical VC approach has been significantlyof the most popular approaches to VC is based on statisticalprocessing, which is capable of extracting a complex conversion improved by the use of a probabilistic conversion function
function in a data-driven manner. Although this technique was [2] and its extension to a trajectory-wise conversion function
originally studied in the context of speaker conversion, which capable of effectively reproducing natural speech parameter
converts the voice of a certain speaker to sound like that trajectories [3]. Moreover, its real-time implementation hasof another specific speaker, it has great potential to achieve also been achieved recently [4]. The real-time statistical VC
various applications beyond speaker conversion. This paper
briefly reviews a trajectory-based conversion method that is processing enables us to produce more varieties of speechcapable of effectively reproducing natural speech parameter tra- sounds beyond the physical constraints. This technique tojectories utterance by utterance and highlights several techniques augment the speech production mechanism has great potentialthat extend this trajectory-based conversion method to achieve to develop various applications to break down the existing
real-time conversion processing. Finally this paper shows some barriers in human-to-human speech communication.
examples of real-time VC applications to enhance human-toThis paper presents a real-time statistical VC technique andhuman speech communication, such as speaking-aid, silent speech
its applications for the augmented speech production. Aftercommunication, and voice changer/vocal effector.
Index Terms—human-to-human speech communication en- briefly reviewing basic VC frameworks in Section II, we
hancement, augmented speech production, statistical voice con- describe a trajectory-based conversion method using a Gausversion, real-time processing
sian mixture model (GMM) [3] as one of the well-formulated
I. I NTRODUCTION
probabilistic conversion methods in Section III and its realIn human-to-human speech communication, we can simul- time implementation [4] by highlighting techniques to achievetaneously convey not only linguistic information but also para- a lower conversion delay and efficient conversion processing inand non-linguistic information to others. Although lots of Section IV. Finally, this paper shows some applications using
information is encoded on a speech signal, we can immediately the augmented speech production based on the real-time VC
generate it by using speech organs. In a speech production technique in Section V and future directions in Section VI.
mechanism, varieties of voice quality generated by individuals
II. BASIC VOICE C ONVERSION F RAMEWORKS
are limited due to physical constraints. They give a part ofacoustic cues on speaker identity to speech sounds. On the
The output of VC processing is a speech signal conveying
other hand, they also easily cause several barriers in our intended non-/para-linguistic information. On the other hand,speech communication; e.g., severe vocal disorders are caused there are some varieties on the input of VC processing.
even if speech organs are partially damaged. Moreover, therequirement of producing speech sounds enough audible to A. Speech Input
listeners sometimes causes other barriers in some specificenvironments: e.g., we hesitate to talk about something privateusing a cell phone if we are surrounded by others. Thesebarriers would be overcome if our speech production wasaugmented so that we could produce speech sounds as wewant beyond the physical and environmental constraints.
Voice conversion (VC) is a technique for modifying speech
acoustics, converting non-/para-linguistic information to any
form we want while preserving the linguistic information. Oneof the VC frameworks is a conversion of an input speech signal

978-1-4799-7088-9/14/$31.00 ©2014 IEEE

One of the most popular VC frameworks is VC processing
using only a speech signal as the input; i.e., the mapping fromthe source voice parameters to the target voice parameters
is modeled. A mainstream is the statistical approach [1] todetermining the conversion function using a parallel speech
data set consisting of utterance pairs of the source and target
voices. Various methods have been proposed to define theconversion function to capture nonlinearity in the mappinge.g., using GMMs [2], [3], Gaussian process regression [5],[6], neural networks [7], restricted Boltzmann machines [8],755GlobalSIP 2014: Machine Learning Applications in Speech Processing[9], bidirectional associative memories [9], dynamic frequency
warping [10], and non-negative matrix factorization [11], [12].
Because no text transcription is necessary, real-time processing is achievable by using a frame-wise conversion function. As the real-time processing is essential to develop VC
applications in human-to-human speech communication, thispaper focuses on this VC framework. Moreover, languageindependent processing, such as cross lingual VC processing[13], is also implemented easily. On the other hand, mostof the literatures focus on spectral conversion rather thanprosodic conversion. It is not straightforward to precisely
convert prosodic features without any linguistic informationin particular in real-time processing.
B. Text Input
In the context of Text-to-Speech (TTS), which is a techniqueto model the mapping from linguistic features to the target
voice parameters, speech generation processing capable offlexibly controlling non-/para-linguistic information may also
be regarded as VC processing. Recently, a speech synthesis
method based on a hidden Markov model (HMM) has been
widely studied as the most popular statistical parametricspeech synthesis method [14], [15]. VC processing can beeasily achieved by modifying HMM parameters using wellstudied model adaptation techniques, such as maximum aposteriori [16], maximum likelihood linear regression [17],eigenvoices [18], average voice models [19], and multiple
regression models [20].
Thanks to the use of text information, it is straightforward
to control not only spectral features but also prosodic features.
This VC framework is essential in man-to-machine speechcommunication. On the other hand, it is not straightforward to
apply this VC framework to human-to-human speech communication as the text input generally needs more latency than
the speech input.
C. Speech and Text Input
The VC framework using both a speech signal and acorresponding text as the input is capable of using the mappingfrom the source voice parameters given linguistic features
to the target voice parameters. This VC framework is often
used as post-processing of TTS to achieve varieties of theoutput voice quality and speaking styles. Context-dependent
conversion functions are available to convert not only spectral
but also prosodic parameters of a generated speech signal[21]–[24]. This VC framework can be applied to any TTS
system including a concatenative speech synthesizer.
This VC framework is also useful to manually design speaking voices or singing voices by specific target speakers/singers
using a TTS system or a singing voice synthesizer. One
typical method is VocaListener [25] in singing voice synthesis,
which is capable of automatically optimizing input manualparameters of a singing voice synthesizer using singing voices
sung by a user. The VC method using HMM for speaking
voices [26] may also be used for this purpose.

III. T RAJECTORY-BASED S TATISTICAL VC
A. Feature Extraction
Let xt and y t be a D(x) -dimensional speech parameter
vector of the source voice and a D(y) -dimensional speechparameter vector of the target voice, respectively, where t isframe index.1 The speech parameters over multiple frames arelinearly or nonlinearly projected onto feature spaces effectively
capturing their time evolution over an utterance.
As the source feature, a D(X) -dimensional segment feature
vector X t is extracted at each frame from a joint vector developed by concatenating the source speech parameter vectors
over several frames from t − C to t + C as follows:
⊤

⊤
⊤
+ b,
(1)
X t = A x⊤
t−C , · · · , xt , · · · , xt+C

where ⊤ denotes transposition of the vector. The transformation matrix A and the bias vector b are designed according toa type of the source voice.2
As the target feature, a joint static and dynamic feature
vector Y t is extracted at each frame as follows:⊤ ⊤, ∆y t = y t − y t−1 .
(2)
Y t = y⊤t , ∆y t
Moreover,the i global variance (GV) vector v (y ) =
h
⊤
(y )
(y )
is also extracted as a nonlinearly projected
v1 , · · · , vD(y)feature from a time sequence of the target speech parameter
vectors {y 1 , · · · , y T } over an utterance [3] as follows:
!2
T
T
1 X
1X
(y )
yt,d −yτ,d ,
(3)
vd =
T t=1
T τ =1

where yt,d is the dth dimensional component of y t .

B. Training
Using the parallel training data set consisting of utterance
pairs of the source and target voices, the joint source and target
feature vector is developed at each frame by performing timealignment. Then, the joint probability density function (p.d.f.)of the source and target feature vectors is modeled with a
GMM [28] as follows:
P X t , Y t |λ(X,Y )  " (X) #  (XX) (XY ) !
M
X
Σm
Σm
X t µm
,
(4)
αm N
=
;
(Y X)
Y)
) ,
Y t µ(Y
Σ
Σ(Y
m
m
m
m=1

where the Gaussian distribution with a mean vector µ
and a covariance matrix Σ is denoted as N (·; µ, Σ).
The mixture component index is m and the total number of mixture components is M . A parameter set of
the GMM is denoted as λ(X,Y ) , which oconsists of
n
(Y )
(X)
(XX)
(Y Y )(XY )(Y X)
of indiαm , µ m , µ m , Σ m , Σ m , Σ m , Σ mvidualmixturecomponents.
The
p.d.f.
oftheGV
vector
P v (y ) |λ(v) is also modeled using a Gaussian distribution,
where λ(v) denotes its parameter set.

1 It depends on individual VC applications which kinds of speech parameters
are used for the source and target voices.
2 For instance, regression coefficients to calculate dynamic features [3] oreigenvectors to efficiently model the joint vector [27] are used.

756

GlobalSIP 2014: Machine Learning Applications in Speech Processing

C. Conversion
i⊤
i⊤
h
h
⊤
⊤
⊤
,
·
·
·
,
Y
and
Y
=
Y
,
·
·
·
,
X
Let X = X ⊤T1
T1
be a time sequence vector of the source features and thatof the target features, respectively. A timeh sequence vectori⊤of the converted speech parameters ŷ = ŷ ⊤1 , · · · , ŷ Tdetermined as follows: ωŷ = argmax P Y |X, λ(X,Y ) P v (y ) |λ(v) ,

⊤

is

(5)

y

subject to Y = W y,
(y)

(6)

(y)

where W is the 2D T -by-D T matrix to extend a time
sequence vector of the static feature vectors into that of
the joint static and dynamic feature vectors [29]. The GV
likelihood weight is ω.
Trajectory-wise batch-type processing is required to determine ŷ. To reduce the computationalcost in this process(X,Y )
modeled by theing, the conditional p.d.f. P Y |X, λ
GMM is often approximated with a single mixture component
sequence m̂ = {m̂1 , · · · , m̂T }, which is determined asarg max P (m|X t , λ(X,Y ) ).
(7)
m


Then, the conditional p.d.f. P Y t |X t , m̂t , λ(X,Y ) at eachframe is modeled by a Gaussian distribution of which mean
vector and covariance matrix are given by(X)(Y X) (XX) −1(Y )(Y |X)
X t − µm̂t , (8)= µm̂t + Σm̂t Σm̂t
µm̂t ,t
m̂t

=

=

Σm̂t

(Y |X)Σm̂t(Y Y )(Y X)− Σm̂t(XX) −1

Σm̂t

(XY )

Σm̂t

,

IV. R EAL -T IME I MPLEMENTATION OF
T RAJECTORY-BASED S TATISTICAL VC
A. From Trajectory-Wise to Frame-Wise Processing
1) Low-Delay Conversion: Trajectory-wise processing todetermine the initial estimate of the converted speech parameter sequence ignoring the GV p.d.f. can be well approximated with frame-wise processing using the low-delay
conversion algorithm [30]. The time sequence vector y d is
divided into overlapped (L + 1)-dimensional segment vectors(t)⊤y d = [yt−L,d , · · · , yt,d ] at individual frames. This segment
vector is recursively updated frame by frame, and then, the first(t)component of the updated segment vector ŷ d , i.e., ŷt−L,d ,is determined as the dth dimensional component of ŷ t−L .
This frame-wise determination processing can be describedby Kalman filtering for the following linear dynamical system(t)using the segment vector y d as a state space parameter:
i⊤
h
(t)
(t−1)
(y|X)
(y|X)
yd = J L yd
+ 01×L , µm̂t ,t,d + nm̂t ,t,d , (10)(∆y|X)(t)(∆y|X)(Y |X)

(11)

(y|X)

tions N nm̂t ,t,d ; 0, Σm̂t ,d(y|X)
Σm̂t ,d(∆Y |X)(∆y|X)and N nm̂t ,t,d ; 0, Σm̂t ,d(∆y|X)
Σm̂t ,d,dth -respectively, whereandare theand(Y |X)(y) th,(d + D ) -dimensional diagonal components of Σmrespectively. The (L + 1)-dimensional row vector wL and the(L + 1) × (L + 1) matrix J L are given by
0 I L×L
wL = 01×(L−1) , −1, 1 , J L =,(12)
0 01×L

where I L×L is a L × L identity matrix. In Kalman filtering,(t)
the posterior
p.d.f. pao
n p.d.f. of y d given the conditional(∆y|X)(y|X)(∆y|X)(y|X)
is recursively
rameters µm̂t ,t,d , µm̂t ,t,d , Σm̂t ,d , Σm̂t ,destimated frame by frame and its mean vector is regarded as(t)the estimate of the segment vector ŷ d . Then, the posterior
p.d.f. is propagated to next frame. This low-delay conversion
processing enables to approximately determine the converted
speech parameter vector ŷ t frame by frame considering all
past frames, a current frame, and next L frames.
2) Postfiltering with Global Variance: To approximately
determine the converted speech parameter vectors also considering the GV p.d.f., their initial estimates determined by
Kalman filtering are modified by postfiltering based on a
global linear transformation, which is given by(9)respectively. Off-diagonal components of the covariance ma(Y |X)
are also often ignored to reduce the computationaltrix Σm̂t
cost. First, further ignoring the GV p.d.f., an initial estimate ofa time sequence vector ŷ d consisting of the dth dimensional
components of ŷ is analytically determined [29]. Then, it isfurther iteratively updated also considering the GV p.d.f. [3].

µm̂t ,t,d = wL y d + nm̂t ,t,d ,(y|X)

where 0N ×M is a N × M zero matrix, and µm̂t ,t,d(∆y|X)and µm̂t ,t,d are the dth - and (d + D(y) )th -dimensional(Y |X)
components of µm,t , respectively. The random variables(Y |X)(∆Y |X)nm̂t ,t,d and nm̂t ,t,d are followed by Gaussian distribu-

(GV)

ŷt,d

1

(v) 2

= µd

1

(v)− 2µ̂d(ŷt,d − hŷd i) + hŷd i ,

(13)

(v)

where µd is the dth dimensional
componentof the mean(y ) (v). On the other hand,
vector of the GV p.d.f. P v |λ(v)

µ̂d and hŷd i are the dth dimensional components of the GV
vector and a mean vector of the converted speech parameters
before postfiltering. They are approximated with their meanvalues calculated in advance from the target speech parametersconverted from the source feature vectors in training data withthe trained GMM. This frame-by-frame postfiltering causes nosignificant degradation in converted speech quality comparedwith the batch-type trajectory-based VC processing [4].
B. Reduction of Computational Cost
1) Computationally Efficient Source Feature Extraction:
Most of the VC applications need to use a spectral parameterfor the source feature extraction. To significantly reduce thecomputational cost of spectral analysis, simple FFT analysis
is used to extract a low-quality spectral parameter. On theother hand, for the target feature extraction, the state-of-theart analysis methods, such as STRAIGHT analysis [31], areused to extract high-quality speech parameters. Then, theconversion function from the low-quality source feature to thehigh-quality target features is trained using these feature pairs.
This technique can significantly reduce the computational costin conversion while causing no adverse effect on converted
speech quality [4].

757GlobalSIP 2014: Machine Learning Applications in Speech Processing2) Rapid mixture component selection: In some VC applications, full covariance matrices need to be used because
different types of speech parameters are used as the source
and target features, e.g., a spectral parameter for the source
and an F0 parameter for the target. It significantly increases
the computational cost for the mixture component selection in
Eq. (7). To reduce it while keeping the conversion accuracy
high enough, diagonalization of the mixture-dependent fullcovariance matrices using a global linear transform, whichis also known as the semi-tied covariance [32], is effective.
are approximated with
The source covariance matrices Σ(XX)
m
′
X′) ⊤
R
,
where
R
is
the
globallinear transform and
RΛ(X
m
′
X′)
is
the
mixture-dependent
diagonal
matrices, enablingΛ(Xmto use the diagonal matrices in Eq. (7). Thanks to thistechnique, the real-time statistical VC processing has beensuccessfully implemented for a floating point digital signal
processor (DSP), TMS320C6748 (375 MHz) [33].
V. A PPLICATIONS FOR AUGMENTED S PEECH P RODUCTION
A. Alaryngeal Speech Enhancement for Speaking-Aid
Laryngectomees need to use alternative speaking methods,
such as an esophageal speaking method or a speaking method
using an electrolarynx, to produce sound source signals because they have lost their own vocal codes. Although thesemethods are very helpful for laryngectomees to produce
speech sounds again, the produced speech (called alaryngeal
speech) sounds very unnatural and less intelligible comparedwith natural voices, causing degradation of quality of life.
To address this issue, we have proposed alaryngeal speechenhancement methods based on the real-time statistical VC
technique to convert alaryngeal speech into natural voices. The
spectral segment feature of the alaryngeal speech is used as
the source feature and spectral and excitation parameters of thenatural voice are used as the target features. The GMMs are
separately trained for the source and individual target featurepairs. Moreover, to allow laryngectomees to control enhanced
voice quality as they want (e.g., to recover their own lost voice
quality), we have also applied eigenvoice conversion (EVC)
techniques [34] to the alaryngeal speech enhancement [35].
Our experimental results have revealed that the alaryngeal
speech enhancement can significantly improve naturalness ofalaryngeal speech; on the other hand, it tends to cause slightdegradation in intelligibility for EL speech. To address thisissue, we have also developed a hybrid approach and haveconfirmed its effectiveness [36].
B. Body-Conducted Speech Enhancement for Silent SpeechCommunication
Recently silent speech interfaces have attracted attention asa technology to support new speech communication styles,
enabling speech communication to take place without thenecessity of emitting an audible acoustic signal [37]. As one ofthem, we focus on the use of a body-conductive microphone, inparticular nonaudible murmur (NAM) microphone [38]. NAM
microphone has been designed to detect a very soft whispered
voice, and therefore, it allows us to speak less audible speech.
However, the detected speech signal sounds unnatural and lessintelligible due to a body-conductive recording mechanism. To
make it possible to use such an unnatural and less intelligible
speech in human-to-human speech communication, we haveproposed body-conducted speech enhancement methods basedon the real-time statistical VC technique to convert the bodyconducted speech into air-conducted speech and have reported
that naturalness and intelligibility are significantly improvedby these enhancement methods [39].
C. Voice Changer and Vocal Effector
The real-time statistical VC enables us to produce various
varieties of voice quality beyond the physical constraints.
By further applying EVC techniques [34] to the real-timestatistical VC, we have developed a new type of voice changer
allowing us to convert our own voice not only to the specifictarget speaker’s voice but also to various voice quality definedby a small amount of manually controllable parameters [40].
To bring a new singing expression, we have also developed
vocal effector by applying the real-time statistical VC based on
EVC techniques to singing VC processing [41]. This allows us
to sing with varieties of singing voice quality. It has also been
extended to control the perceived age of the singing voices
while preserving singer’s identity [42].
VI. T OWARDS F URTHER I MPROVEMENTS
Performance of the real-time statistical VC still needs tobe improved. One direction is to apply better p.d.f. modelingtechniques, such as deep neural networks. There have been
several attempts at applying them to VC frameworks. Chen etal. [9] have reported that 1) their effectiveness is observed if
directly modeling spectral envelop parameters without using
any specific parameterization, such as mel-cepstrum, and 2)
the generative training is more effective than the discriminative
training, such as fine-tuning using back propagation withminimum mean square error. It will be essential to find an
optimization criterion consistent with human perception.
Another direction is to find features not well reproduced in
the mapping functions. The conversion process shown in Eq.
(5) can be regarded as a framework using product of experts[43] and the GV expert effectively works as a penalty term
to alleviate the over-smoothing effect often observed in thestatistical VC framework. Recently, Takamichi et al. [44] havefound that a metric on the modulation spectrum of spectral
parameters detect the over-smoothing effect more sensitively
than that on the GV. It is expected that further improvements
will be yielded by incorporating such a feature as a new expert.
In some applications of the augmented speech production,
F0 pattern prediction is also essential. It will be helpful to not
only use better prediction models but also integrate better F0
pattern representations, such as the F0 generative model [45].
We also have to look at a possibility that the real-timestatistical VC is misused for spoofing, also keeping it in mind
that this technique has a great potential to break down several
barriers in human-to-human speech communication. We needto tell people how to use it correctly like a kitchen knife.
ACKNOWLEDGEMENT: Part of this work was supported by JSPS KAKENHI Grant Number 26280060.

758

GlobalSIP 2014: Machine Learning Applications in Speech Processing

R EFERENCES[1] M. Abe, S. Nakamura, K. Shikano, and H. Kuwabara. Voice conversion
through vector quantization. J. Acoust. Soc. Jpn. (E), Vol. 11, No. 2, pp.
71–76, 1990.
[2] Y. Stylianou, O. Cappé, and E. Moulines. Continuous probabilistictransform for voice conversion. IEEE Trans. Speech & Audio Process.,
Vol. 6, No. 2, pp. 131–142, 1998.
[3] T. Toda, A.W. Black, and K. Tokuda. Voice conversion based on
maximum likelihood estimation of spectral parameter trajectory. IEEE
Trans. Audio, Speech & Lang. Process., Vol. 15, No. 8, pp. 2222–2235,
2007.
[4] T. Toda, T. Muramatsu, and H. Banno. Implementation of computationallyefficient real-time voice conversion. Proc. INTERSPEECH, Portland,
USA, Sep. 2012.
[5] N. Pilkington, H. Zen, and M.J.F. Gales. Gaussian process experts forvoice conversion. Proc. INTERSPEECH, pp.2761–2764, Florence, Italy,
Aug. 2011.
[6] N. Xu, Y. Tang, J. Bao, A. Jiang, X. Liu, and Z. Yang. Voice conversion
based on Gaussian processes by coherent and asymmetric training withlimited training data. Speech Communication Vol. 58, pp. 124–138, 2014.
[7] M. Narendranath, H. A. Murthy, S. Rajendran, and B. Yegnanarayana.
Transformation of formants for voice conversion using artificial neuralnetworks. Speech Communication, Vol. 16, No. 2, pp. 207–216, 1995.
[8] T. Nakashika, T. Takiguchi, and Y. Ariki. Voice conversion based on
speaker-dependent restricted Boltzmann machines. IEICE Trans. Inf. &Syst., Vol. E97-D, No. 6, pp. 1403–1410, 2014.
[9] L.-H. Chen, Z.-H. Ling, L.-J. Liu, and L.-R. Dai. Voice conversion using
deep neural networks with layer-wise generative training. IEEE/ACM
Trans. Audio, Speech, & lang. Process., Vol. 22, No. 12, pp. 1859–1872,
2014.
[10] D. Erro, A. Moreno, and A. Bonafonte. Voice conversion based onweighted frequency warping. IEEE Trans. Audio, Speech & Lang.
Process., Vol. 18, No. 5, pp. 922–931, 2010.
[11] R. Takashima, T. Takiguchi, and Y. Ariki. Exemplar-based voice
conversion using sparse representation in noisy environments. IEICE
Trans. Fundamentals, Vol. E96-A, No. 10, pp. 1946–1953, 2013.
[12] Z. Wu, T. Virtanen, E.S. Chng, and H. Li. Exemplar-based sparse representation with residual compensation for voice conversion. IEEE/ACM
Trans. Audio, Speech, & lang. Process., Vol. 22, No. 10, pp. 1506–1521,
2014.
[13] M. Abe, K. Shikano, and H. Kuwabara. Statistical analysis of bilingual
speaker’s speech for cross-language voice conversion. J. Acoust. Soc.
Am., Vol. 90, No. 1, pp. 76–82, 1991.
[14] K. Tokuda, Y. Nankaku, T. Toda, H. Zen, J. Yamagishi, and K. Oura.
Speech synthesis based on hidden Markov models. Proceedings of theIEEE, Vol. 101, No. 5, pp. 1234–1252, 2013.
[15] H. Zen, K. Tokuda, and A.W. Black. Statistical parametric speechsynthesis. Speech Communication, Vol. 51, No. 11, pp. 1039–1064, 2009.
[16] J.-L. Gauvain and C.-H. Lee. Maximum a posteriori estimation for
multivariate Gaussian mixture observations of Markov chains. IEEE
Trans. Speech & Audio Process., Vol. 2, No. 2, pp. 291–298, 1994.
[17] M.J.F. Gales. Maximum likelihood linear transformations for HMMbased speech recognition. Computer Speech & Language, Vol. 12, No.
2, pp. 75–98, 1998.
[18] R. Kuhn, J. Junqua, P. Nguyen, and N. Niedzielski. Rapid speaker
adaptation in eigenvoice space. IEEE Trans. Speech & Audio Process.,
Vol. 8, No. 6, pp. 695–707, 2000.
[19] J. Yamagishi and T. Kobayashi. Average-voice-based speech synthesis
using HSMM-based speaker adaptation and adaptive training IEICE
Trans. Inf. & Syst., Vol. E90-D, No. 2, pp. 533–543, 2007.
[20] T. Nose, J. Yamagishi, T. Masuko, and T. Kobayashi. A style control
technique for HMM-based expressive speech synthesis. IEICE Trans. Inf.
& Syst., Vol. E90-D, No. 9, pp. 1406–1413, 2007.
[21] C.-H. Wu, C.-C. Hsia, T.-H. Liu, and J.-F. Wang. Voice conversion using
duration-embedded bi-HMMs for expressive speech synthesis. IEEE
Trans. Audio, Speech, & Lang. Process., Vol. 14, No. 4, pp. 1109–1116,
2006.
[22] J. Tao, Y. Kang, and A. Li. Prosody conversion from neutral speech toemotional speech. IEEE Trans. Audio, Speech, & Lang. Process., Vol.
14, No. 4, pp. 1145–1154, 2006.
[23] C.-H. Wu, C.-C. Hsia, C.-H. Lee, and M.-C. Lin. Hierarchical prosody
conversion using regression-based clustering for emotional speech synthesis. IEEE Trans. Audio, Speech, & Lang. Process., Vol. 18, No. 6, pp.
1394–1405, 2010.

[24] Z. Inanoglu and S. Young. Data-driven emotion conversion in spoken
English. Speech Communication, Vol. 51, No. 3, pp. 268–283, 2009.
[25] T. Nakano and M. Goto. VocaListener: a singing-to-singing synthesis
system based on iterative parameter estimation. Proc. 6th Sound & Music
Computing Conference, pp. 343–348, Porto, Portugal, July 2009.
[26] T. Nose, Y. Ota, and T. Kobayashi. HMM-based voice conversion using
quantized F0 context. IEICE Trans. Inf. & Syst., Vol. E93-D, No. 9, pp.
2483–2490, 2010.
[27] T. Toda, A.W. Black, and K. Tokuda. Statistical mapping between
articulatory movements and acoustic spectrum using a Gaussian mixture
model. Speech Communication, Vol. 50, No. 3, pp. 215–227, 2008.
[28] A. Kain and M. W. Macon. Spectral voice conversion for text-to-speechsynthesis. Proc. ICASSP, Seattle, U.S.A., pp. 285–288, May 1998.
[29] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura.
Speech parameter generation algorithms for HMM-based speech synthesis. Proc. ICASSP, pp. 1315–1318, Istanbul, Turkey, June 2000.
[30] T. Muramatsu, Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano.
Low-delay voice conversion based on maximum likelihood estimationof spectral parameter trajectory. Proc. INTERSPEECH, pp. 1076–1079,
Brisbane, Australia, Sep. 2008.
[31] H. Kawahara, I. Masuda-Katsuse, and A. de Cheveigné. Restructuring
speech representations using a pitch-adaptive time-frequency smoothing
and an instantaneous-frequency-based F0 extraction: possible role of arepetitive structure in sounds. Speech Communication, Vol. 27, No. 3–4,
pp. 187–207, 1999.
[32] M.J.F. Gales. Semi-tied covariance matrices for hidden Markov models.
IEEE Trans. Speech & Audio Process., Vol. 7, No. 3, pp. 272–281, 1999.
[33] T. Moriguchi, T. Toda, M. Sano, H. Sato, G. Neubig, S. Sakti,and S. Nakamura.
A digital signal processor implementation ofsilent/electrolaryngeal speech enhancement based on real-time statisticalvoice conversion. Proc. INTERSPEECH, pp. 3072–3076, Lyon, France,
Aug. 2013.
[34] T. Toda, Y. Ohtani, and K. Shikano. One-to-many and many-to-one
voice conversion based on eigenvoices. Proc. ICASSP, pp. 1249–1252,
Hawaii, USA, Apr. 2007.
[35] H. Doi, T. Toda, K. Nakamura, H. Saruwatari, and K. Shikano. Alaryngeal speech enhancement based on one-to-many eigenvoice conversion.
IEEE/ACM Trans. Audio, Speech & Lang. Process., Vol. 22, No. 1, pp.
172–183, 2014.
[36] K. Tanaka, T. Toda, G. Neubig, S. Sakti, and S. Nakamura. A
hybrid approach to electrolaryngeal speech enhancement based on noise
reduction and statistical excitation generation. IEICE Trans. Info. & Syst.,
Vol. E97-D, No. 6, pp. 1429–1437, 2014.
[37] B. Denby, T. Schultz, K. Honda, T. Hueber, J.M. Gilbert, and J.S.
Brumberg. Silent speech interfaces. Speech Communication, Vol. 52,
No. 4, pp. 270–287, 2010.
[38] Y. Nakajima, H. Kashioka, N. Cambell, and K. Shikano. Non-Audible
Murmur (NAM) recognition. IEICE Trans. Inf. & Syst., Vol. E89-D, No.
1, pp. 1–8, 2006.
[39] T. Toda, M. Nakagiri, and K. Shikano. Statistical voice conversion
techniques for body-conducted unvoiced speech enhancement. IEEE
Trans. Audio, Speech & Lang. Process., Vol. 20, No. 9, pp. 2505–2517,
Sep. 2012.
[40] K. Ohta, T. Toda, Y. Ohtani, H. Saruwatari, and K. Shikano. Adaptive
voice-quality control based on one-to-many eigenvoice conversion. Proc.
INTERSPEECH, pp. 2158–2161, Chiba, Japan, Sep. 2010.
[41] H. Doi, T. Toda, T. Nakano, M. Goto, and S. Nakamura. Singing voice
conversion method based on many-to-many eigenvoice conversion andtraining data generation using a singing-to-singing synthesis system. Proc.
APSIPA ASC, Hollywood, USA, Nov. 2012.
[42] K. Kobayashi, T. Toda, H. Doi, T. Nakano, M. Goto, G. Neubig, S.
Sakti, and S. Nakamura. Voice timbre control based on perceived age insinging voice conversion. IEICE Trans. Info. & Syst., Vol. E97-D, No.
6, pp. 1419-1428, 2014.
[43] H. Zen, M.J.F. Gales, Y. Nankaku, and K. Tokuda. Product of experts
for statistical parametric speech synthesis. IEEE Trans. Audio, Speech &Lang. Process., Vol. 20, No. 3, pp. 794–805, 2012.
[44] S. Takamichi, T. Toda, G. Neubig, S. Sakti, and S. Nakamura. A
postfilter to modify the modulation spectrum in HMM-based speechsynthesis. Proc. ICASSP, pp. 290–294, Florence, Italy, 2014.
[45] H. Kameoka, K. Yoshizato, T. Ishihara, Y. Ohishi, K. Kashino, and
S. Sagayama. Generative modeling of speech F0 contours. Proc.
INTERSPEECH, pp. 1826–1830, Lyon, France, Aug. 2013.

759