MODULATION SPECTRUM-CONSTRAINED TRAJECTORY TRAINING ALGORITHM
FOR GMM-BASED VOICE CONVERSION
Shinnosuke Takamichi†,‡ , Tomoki Toda† , Alan W. Black‡ , and Satoshi Nakamura†† Graduate School of Information Science, Nara Institute of Science and Technology (NAIST), Japan‡ Language Technologies Institute, Carnegie Mellon University (CMU), U.S.A
Email: shinnosuke-t@is.naist.jp
ABSTRACTThis paper presents a novel training algorithm for Gaussian Mixture
Model (GMM)-based Voice Conversion (VC). One of the advantagesof GMM-based VC is computationally efficient conversion processing enabling to achieve real-time VC applications. On the other
hand, the quality of the converted speech is still significantly worse
than that of natural speech. In order to address this problem while
preserving the computationally efficient conversion processing, theproposed training method enables 1) to use a consistent optimization criterion between training and conversion and 2) to compensate
a Modulation Spectrum (MS) of the converted parameter trajectory
as a feature sensitively correlated with over-smoothing effects causing quality degradation of the converted speech. The experimental results demonstrate that the proposed algorithm yields significant
improvements in term of both the converted speech quality and theconversion accuracy for speaker individuality compared to the basic
training algorithm.
Index Terms— GMM-based voice conversion, over-smoothing,modulation spectrum, training algorithm
1. INTRODUCTION
Statistical Voice Conversion (VC) is an effective technique for modifying speech parameters to convert non-linguistic information while
keeping linguistic information unchanged, and making it possible toenhance various speech-based systems [1, 2, 3, 4]. Recently, several
state-of-the-art methods have been applied to VC [5, 6, 7], but Gaussian Mixture Model (GMM)-based VC [8, 9] has still gained popularity thanks to its computationally efficient conversion processing.
This framework models the joint probability density of the static anddynamic features of speech parameters from both source and target
voices using a GMM. The conversion stage performs the Maximum
Likelihood (ML)-based trajectory conversion [9] using the conditional probability density analytically derived from the GMM and an
additional constraint between the static and dynamic features. In thismethod, the converted parameter trajectories can be determined analytically, also enabling the computationally efficient real-time conversion processing [10, 11]. However, they suffer from the oversmoothing effect, which makes the converted speech sound muffled.
There are various attempts at addressing the over-smoothingissue in statistical parametric speech synthesis. Zen et al. [12]proposed the trajectory training method for Hidden Markov Model(HMM)-based speech synthesis [13] by imposing the constraint between the static and dynamic features in synthesis on the training
criterion. The use of a consistent optimization criterion between
training and synthesis enables to effectively optimize the trainedmodel for synthesis. Toda and Young [14] has further incorporated
an additional criterion on Global Variance (GV) into the optimization criterion, where the GV is well known as a feature sensitively

978-1-4673-6997-8/15/$31.00 ©2015 IEEE

4859

correlated to the over-smoothing effect [9]. This method enables
the use of the computationally efficient parameter generation algorithm [15] for generating the parameter trajectory while keeping
its GV close to natural one. Hwang et al. [16] proposed the training algorithm for GMM-based VC considering the GV. Although
it significantly improves the quality of the converted speech while
keeping the computationally efficient conversion, there still remainsome problems to be solved: 1) it suffers from the inconsistent optimization criteria between training and conversion, and 2) the GV isstill insufficient to detect the over-smoothing effect.
Recently, Takamichi et al. [17] have found that a Modulation
Spectrum (MS) of the generated parameter trajectory is more sensitively correlated to the over-smoothing effect than the GV. They
have also reported that the MS can be regarded as an extension of
the GV and the synthetic speech quality is significantly improved byconsidering the MS rather than the GV in HMM-based speech synthesis. It is expected that a better consistent optimization criterion
will be designed by incorporating a criterion on the MS.
This paper proposes an MS-constrained trajectory training algorithm as a novel training algorithm for GMM-based VC. After
implementing the trajectory training, we further integrate the MS
into the trajectory training. The proposed training algorithm gives aunified framework for both training and conversion which provides
both a consistent optimization criterion and a closed form solution
for parameter conversion considering the MS. The experimental results demonstrate that the proposed algorithm yields significant improvements in term of both speech quality and speaker individuality.
2. BASIC FRAMEWORK
2.1. GMM Training [8]A joint probability density function of speech parameters of thesource and target speakers is modeled with a GMM using a parallel
data as follows:
] ) ∑
]
)
([
([
M
Xt
Xt
)
(X,Y ); µ(X,Y
,
Σ
, (1)
P
λ =
αm N
m
m
Yt
Yt
m=1
[
]
[ (X) ]
(XX)
(XY )
µm
Σm
Σm
(X,Y )(X,Y )
µm
=
,
Σ
=
, (2)
m
)
(Y X)
(Y Y )µ(YΣmΣmm
where X t and Y t are the source and target feature vectors at framet, respectively. Y t is given by 2D-dimensional joint static and dy[]⊤ ⊤namic feature vectors, y ⊤, where y t is represented as at , ∆y t
D-dimensional vector, [yt (1) , · · · , yt (D)]⊤ . The source feature
vector is also given by the same form in this paper. N (·; µ, Σ) denotes the Gaussian distribution with a mean vector µ and a covariance matrix Σ. The total number of mixture components is M . λ isa GMM parameter set consisting of the mixture-component weightICASSP 2015

(X,Y ))
αm , the mean vector µ(X,Yand the covariance matrix Σmofm)the m-th mixture component. µ(X,Yconsists of the source and tarm
(X,Y )(Y )get mean vectors, µ(X)consists of the source andm and µm . Σm
(XX)
(Y Y )
target covariance matrices, Σmand Σmand cross-covariance
(Y X)
(XY )
matrices, Σmand Σm , where they are diagonal matrices inthis paper. The GMM parameters are estimated by maximizing thejoint probability density given by Eq. (1).

3.3. Parameter Conversion
The basic parameter conversion process is performed as describedin Section 2.2. The converted parameter sequence ŷ m̂ is equivalent
to that determined by maximizing Lgv under the constraint Y =W y. Therefore, Lgv is also regarded as the objective function forconversion although the constraint needs to be additionally used.
3.4. Problems

2.2. Parameter Conversion [9]][⊤ ⊤Given the T -frame feature sequence X = X ⊤of the1 , · · · , XT
source speaker’s voice, the converted parameter sequence ŷ m̂ =[ ⊤]⊤ŷ 1 , · · · , ŷ ⊤is analytically determined by maximizing the conT
ditional probability of the target feature vector Y given X under aconstraint Y = W y as follows:(3)ŷ m̂ = argmax P (W y|X, m̂, λ)
y

=

R−1
m̂ r m̂ =()−1
W ⊤ D −1W ⊤ D −1m̂ Wm̂ E m̂ , (4)where W is a 2DT -by-DT weight matrix to calculate the dynamic
features [15]. m̂ = [m̂1 , · · · , m̂T ]⊤ is the sub-optimum mixture
component sequence determined by maximizing P (m|X, λ), andthe other parameters are given by[]⊤(Y |X) −1(Y |X) −1−1
D m̂= diag2D Σm̂1
, · · · , Σm̂T
, (5)
[
]
⊤
(Y |X) ⊤(Y |X) ⊤E m̂ =µm̂1 ,1 , · · · , µm̂T ,T(6),(Y |X)µm̂t ,t=Am̂ X t + bm̂ ,

Am̂

=

Σm̂

,

(8)

=

(Y )
(Y X) (XX) −1 (X)µm̂ − Σm̂ Σm̂
µm̂ ,(Y Y )(Y X) (XX) −1 (XY )

(9)

bm̂
(Y |X)Σm̂(Y X)(XX) −1

Σm̂

(7)

(10)
= Σm̂
− Σm̂ Σm̂Σm̂ .
The notation diag2D denotes the construction of a block diagonal
matrix that has the 2D-by-2D diagonal elements. The converted parameter sequence ŷ m̂ is efficiently calculated sequence by sequence
using the Cholesky decomposition [15], or recursively calculated
frame by frame using the low-delay conversion algorithm [18].
3. CONVENTIONAL GV-CONSTRAINED TRAINING
3.1. Global Variance (GV) [9]The GV v (y) = [v (1) , · · · , v (D)]⊤ is defined as the second ordermoment of the trajectory y, and its d-th component is given as
(
)2
T
T
1 ∑
1 ∑
yt (d) −yτ (d) .
(11)v (d) =
T t=1
T τ =13.2. Objective Function for GV-Constrained Training [16]There is inconsistency of the optimization criterion between the
training and the conversion; i.e., the explicit relationship between
the static and dynamic features given by Y = W y is ignored intraining while it is considered in conversion. Therefore, the trainedmodel parameters are not optimum for conversion.
The GV likelihood is used as a penalty term to alleviate the oversmoothness. Although it can improve the quality of the converted
speech, the improved quality is still far from the natural one.
4. IMPLEMENTATION OF TRAJECTORY TRAINING
In [19], the trajectory training has been implemented for the joint
probability density modeling in GMM-based voice conversion. In
this section, we present yet another implementation by reformulating
the conditional probability density function in Eq. (3) by imposing
the explicit relationship between the static and dynamic features.
4.1. Objective Function for Training and Conversion
The objective function for the trajectory training is written as:()
Ltrj = P (y|X, m̂, λ) = N y; ŷ m̂ , R−1(14)m̂
The mean vector ŷ m̂ is given by Eq. (4) and the inter-frame correlation is effectively modeled by the temporal covariance matrix
R−1m̂ . In training, the GMM parameters are updated by maximizing
Ltrj . In conversion, the basic conversion process described in Section 2.2 is performed. Note that the mean vector ŷ m̂ is equivalent
to the generated parameter sequence in the basic conversion process.
Therefore, Ltrj can be regarded as the objective function not only
for training but also for conversion.
4.2. Estimation of Model Parameters
[
]
[
]
⊤ ⊤
⊤ ⊤
Here, let ξA = A⊤, ξb = b⊤, and
1 , · · · , AM
1 , · · · , bM[]⊤
(Y |X) −1(Y |X) −1−1
Σ= Σ1, · · · , ΣM
be the joint parameters of(Y |X) −1

Am , bm , and Σmover all mixture components, respectively.
To optimize these model parameters for the objective function, we
employ the steepest descent algorithm1 as follows:∂ log Ltrj
(i+1)
(i)
ξA
= ξA + α
,
(15)
(i)
∂ξA
ξ =ξ
A

(Y |X)A part of the GMM parameter set, {Am , bm , Σm
} is updated

by maximizing the following objective function Lgv consisting of
the GMM and GV likelihoods:
Lgv = P (W y|X, m̂, λ) P (v (y) |X, m̂, λ, λv )2ωv T ,(12)
P (v (y) |X, m̂, λ, λv ) = N (v (y) ; v (ŷ m̂ ) , Σv ) , (13)where ωv is a weight of the GV likelihood, Σv is a covariance matrixof the GV, and λv is a model parameter set of the GV. The GMM
and GV likelihoods are normalized by the ratio of the number offeature dimensions when ωv = 1.0. This training algorithm updates
the model parameters to make the GV of the converted parametersequence close to natural one.

4860

A

where α is a learning rate, and i is an iteration index. ξb and Σ−1
are also optimized in the same manner. The gradients are given by:[]∂ log Ltrj
−1
−1
⊤
= S⊤
(16)
,
m̂ diag2D D m̂ W (y − ŷ m̂ ) X∂ξA∂ log Ltrj
−1
= S⊤
(17)
m̂ D m̂ W (y − ŷ m̂ ) ,
∂ξb
[
(
)
1 ⊤
∂ log Ltrj
−1⊤⊤=
S m̂ diag−1
2D W Rm̂ + ŷ m̂ ŷ m̂ − yy
−1
2
∂Σ
]
−E m̂ (ŷ m̂ − y)⊤ W ⊤ − W (ŷ m̂ − y) E ⊤m̂ ,(18)
1 Closed form solutions also exist for ξ

A and ξb .

where S m̂ = [S m̂1 , · · · , S m̂T ]⊤ ⊗I 2D is a 2DT -by-2DM matrix,
S m̂t is an M -dimensional vector of which the m-th component is 1
when m = m̂t and otherwise are 0, and I 2D indicates the 2D-by2D identity matrix.
In this paper, the traditional joint density training is firstly
performed to estimate λ. Then, the proposed algorithms up(XX)(X)
dates {ξA , ξb , Σ−1 } while keeping {ωm , µm
, Σm } constant.
Note that the sub-optimum mixture component sequence m̂ neverchanges in this setting.

and the gradients of log Lms are given as][∂ log Lms
−1
−1
−1
⊤
= S⊤
,
(26)
m̂ diag2D D m̂ W Rm̂ sm̂ X∂ξA∂ log Lms
−1
−1
= S⊤
(27)
m̂ D m̂ W Rm̂ sm̂ ,∂ξb]∂ log Lms
−1 [
−1
=S ⊤
m̂ diag2D W Rm̂ sm̂ (E m̂ − W ŷ m̂ ) ,(28)
∂Σ−1
where
[
]
⊤
⊤
⊤ ⊤
sm̂ =
s′1 , · · · , s′t , · · · , s′T
,
(29)
s′t

5. PROPOSED MODULATION
SPECTRUM-CONSTRAINED TRAJECTORY TRAINING

st (d)

sd (f )

=
=

2
2
Rd,f
+ Id,f( T∑)2

yt (d) cos kt

+

t=1

( T
∑

)2
yt (d) sin kt

(21)
,(22)

t=1

where 2Ds is a length of Discrete Fourier Transform (DFT), k =−πf /Ds is a modulation frequency, and Ds′ is the number of MS
dimension in each feature dimension, where Ds′ < Ds .2 In thispaper, the MS is calculated utterance by utterance.

5.2. Proposed Objective Function
We integrate the MS compensation into the trajectory training. The
objective function consists of both the trajectory likelihood and theMS likelihood as follows:′
Lmstrj = P (y|X, m̂, λ) P (s (y) |X, m̂, λ, λs )ωs T /Ds , (23)P (s (y) |X, m̂, λ, λs ) = N (s (y) ; s (ŷ m̂ ) , Σs ) , (24)where λs is a model parameter set of the MS, and Σs is a Ds′ D-byDs′ D covariance matrix, and ωs is a weight of the MS likelihood.
The trajectory likelihood and the MS likelihood are normalized by−1
the ratio of the [number of feature dimensions] when ωs = 1. Σs is
(1)
(d)
(D)
(d)
′
represented as ps , · · · , ps , · · · , ps
, where ps is Ds D-by-[st (1) , · · · , st (d) , · · · , st (D)]⊤ ,where R̂d,f and Iˆd,f are calculated using the d-th dimensional components of ŷ m̂ .
5.4. Parameter Conversion
It is unnecessary to consider the MS in parameter conversion because
the GMM parameters are optimized to make the MS of the converted
parameter sequence close to the natural one. Consequently, the basic
parameter conversion algorithm is straightforwardly employed. If
the proposed objective function Lmstrj is used in the parameter conversion, the converted parameter sequence to maximize it is equivalent to ŷ m̂ which is analytically solved. Therefore, the proposed
framework can also be regarded as a unified framework between the
training and conversion process.
Because the MS involves the GV [17], the proposed MSconstrained algorithm well recovers not only the MS but also the GV.
The GV is effectively recovered by the MS-constrained trajectory
training (“MSTRJ”) as observed in Fig. 1.
6. EXPERIMENTAL EVALUATION
6.1. Experimental Conditions
In our experiments, we prepared 2 English speakers (rms and slt)
in the CMU ARCTIC database [22]. We used 50 sentences wereselected for training and remaining 50 sentences for evaluation. We
trained the slt-to-rms GMM. Speech signals were sampled at 16 kHz.
The shift length was set to 5 ms. The 1st-through-24th mel-cepstralcoefficients were used as a spectral parameter and log-scaled F0 and
5 band-aperiodicity [23, 24] were used as excitation parameters. The
STRAIGHT analysis-synthesis system [25] was employed for pa- 

D matrix of which columns correspond to s (d). The MS likelihood
works as a penalty term to the reduction of the temporal fluctuationof the converted parameter sequence.

*() 

' $&5.3. Estimation of Model Parameters
The model parameters are estimated in the same way as the trajectory
training. Let Lms be the MS likelihood N (s (y) ; s (ŷ m̂ ) , Σs ).
The logarithm function of Lmstrj is given byT
log Lmstrj = log Ltrj + ωs ′ log Lms ,(25)
Ds
2 Because lower modulation frequency components mainly affect speech

perception [20, 21], only these components are considered in this paper.

4861

(30)

⊤
2f t (d) ps(d) (s (y) − s (ŷ)) ,

=
(31)
[
( ′
)]⊤
f t (d) = ft,d (0) , · · · , ft,d (f ) , · · · , ft,d Ds − 1, (32)
ft,d (f ) = R̂d,f cos kt + Iˆd,f sin kt,(33)

5.1. Modulation Spectrum
Whereas the GV represents the temporal scaling of the parameter
trajectory as a scalar value in each feature dimension, the MS explicitly represents the temporal fluctuation as a vector [17]. In thispaper, the MS s (y) of the parameter sequence y is defined as[]⊤s (y) =s (1)⊤ , · · · , s (d)⊤ , · · · , s (D)⊤ ,
(19)
)]⊤
[
( ′
, (20)
s (d) =
sd (0) , · · · , sd (f ) , · · · , sd Ds − 1=$  !% #$! " 

2/3

=?>A@

46587

9 .:4;5<7+-,/. 0 1










Fig. 1. An example of the GVs of the converted mel-cepstral coefficients (“nat” indicates natural speech parameter trajectories).

rameter extraction and waveform generation. The spectral parameters and aperiodic components were converted through a 64-mixture
GMM and a 16-mixture GMM, respectively. The log-scaled F0 was
linearly converted. The DFT length to calculate MS was set to 2048
that covers the length of all training utterances. The conventional
GV-constrained training algorithm and the proposed algorithms wereapplied only to spectral component, and the likelihood weight ωv
and ωs were set to 1.0. According to the results of our preliminary
test3 , Ds′ was set to Ds /2 (= 50 Hz).
We compared the following training algorithms:

 
 
$

!%'
"
!%&$

6

687 !$#%
504 $#%
''-13 !".
-120 "-/.
*+, ) ! 



!#"

 


 



69
6




BASIC： Basic training [8]
GV： Conventional training considering the GV [16]TRJ： Proposed trajectory training
MSTRJ： Proposed trajectory training considering the MSThe trajectory likelihood and the MS likelihood for the natural parameter trajectories of the evaluation data were firstly calculated toanalyze the effect of the proposed algorithms. Then, the speech quality and the speaker individuality of the converted speech are evaluated in the perceptual evaluation.

!
:& '(  

   

Fig. 2. Trajectory likelihood for Fig. 3. MS likelihood for the natthe natural parameter trajectories. ural parameter trajectories.
 

 





6.2. Objective Evaluation
Fig. 2 and Fig. 3 illustrate the trajectory likelihood Ltrj and the MS
likelihood Lms for the natural parameter trajectories of the evaluation data, respectively. The trajectory likelihood is normalized by the
total number of frames T . The trajectory training (“TRJ”) slightly
improves the MS likelihood as well as the trajectory likelihood compared to the basic training (“BASIC”). This result shows that the
proposed trajectory training models the parameter trajectories moreaccurately than the basic training.
The MS-constrained trajectory training (“MSTRJ”) well improves the MS likelihood than the other algorithms. This result
demonstrates that the MS of the converted parameter trajectory iswell recovered by “MSTRJ.” It is observed that the trajectory likelihood is significantly degraded by the conventional GV-constrained
training (“GV”). This is because “GV” uses the inconsistent criteria
between training and conversion. This likelihood degradation is
reduced by “MSTRJ.”

'!"

&!

$%"

#$!

%$ &% 
" !!
!

 

# !""
!"










 





Fig. 4. Preference scores on Fig. 5. Preference scores on
speech quality with 95 % confi- speaker individuality with 95 %dence intervals.
confidence intervals.
7. CONCLUSION

6.3. Subjective Evaluation
In the evaluation of the speech quality, a preference test (AB test)
was conducted. We presented every pair of converted speech of 4 algorithms in a random order, and we forced listeners to select speechsample that sounds better quality. Similarity, XAB test on speakerindividuality was conducted using the analysis-synthesized speech
as a reference “X.” 6 listeners participated in each assessment.
The results are illustrated in Fig. 4 and Fig. 5. It is observed that
“TRJ” has higher scores than “BASIC” in term of both the speech
quality and the speaker individuality. On the other hand, the scoresof “TRJ” are lower than “GV.” Therefore, the effect of the GV compensation on the converted speech is larger than that of the trajectory
training. We can see that “MSTRJ” achieves the best scores thanothers in term of the speech quality. This result demonstrates thatthe proposed MS-constrained trajectory training yields the best performance among these training methods.
3 We conducted the preliminary subjective test to investigate the qualitywise effect of higher modulation frequency component of MS. As a result,
there is no significant difference in quality between analysis-synthesized
speech and the speech that MS over 50 Hz was cut.

4862

This paper have proposed novel training algorithms for GMM-based
voice conversion in order to produce the high-quality speech while
preserving the computationally-efficient conversion algorithm. Traditional GMM have been firstly reformulated as the trajectory model
called “trajectory GMM,” then, the Modulation Spectrum (MS) havebeen integrated into the trajectory training. The experimental results
yielded the significant improvements in term of both the speech quality and the speaker individuality of the converted speech. As future
work, we will implement the proposed algorithm for HMM-based
speech synthesis, and apply the proposed algorithm to the voice conversion for arbitrary speakers [26, 27].
Acknowledgements: Part of this work was supported by JSPS
KAKENHI Grant Number 26280060 and Grant-in-Aid for JSPS
Fellows Grant Number 26 · 10354, and part of this work was executed under ”JSPS Strategic Young Researcher Overseas Visits
Program for Accelerating Brain Circulation.”8. REFERENCES[1] K. Tanaka, T. Toda, G. Neubig, S. Sakti, and S. Nakamura. An
evaluation of excitation feature prediction in a hybrid approachto electrolaryngeal speech enhancement. In Proc. ICASSP, pp.
4521–4525, Florence, Italy, May 2014.
[2] K. Kobayashi, T. Toda, T. Nakano, M. Goto, G. Neubig,
S. Sakti, and S. Nakamura. Regression approaches to perceptual age control in singing voice conversion. In Proc. ICASSP,
pp. 7954–7958, Florence, Italy, May 2014.
[3] N. Hattori, T. Toda, H. Kawai, H. Saruwatari, and K. Shikano.
Speaker-adaptive speech synthesis based on eigenvoice conversion and language-dependent prosodic conversion in speechto-speech translation. In Proc. INTERSPEECH, pp. 2769–
2772, Florence, Italy, Aug. 2011.
[4] S. Aryal and R. G.-Osuna. Can voice conversion be used toreduce non-native accents? In Proc. ICASSP, Florence, Italy,
May 2014.
[5] T. Nakashika, T. Takiguchi, and Y. Ariki. Voice conversion intime-invariant speaker independent space. In Proc. ICASSP,
pp. 7939–7943, Frorence, Italy, May 2014.
[6] Z. Wu, T. Virtanen, T. Kinnunen, E. S. Chng, and H. Li.
Exemplar-based voice conversion using non-negative spectrogram deconvolution. In Proc. 8th ISCA SSW, Catalunya, Spain,
Aug. 2013.
[7] E. Helander, H. Silen, T. Virtanen, , and M. Gabbouj. Voice
conversion using dynamic kernel partial least squares regression. IEEE Trans., Vol. 20, No. 3, pp. 806–817, Mar. 2012.
[8] Y. Stylianou, O. Cappe, and E. Moulines. Continuous probabilistic transform for voice conversion. IEEE Trans. Speech
and Audio Processing, Vol. 6, No. 2, pp. 131–142, Mar. 1988.
[9] T. Toda, A. W. Black, and K. Tokuda. Voice conversion basedon maximum likelihood estimation of spectral parameter trajectory. IEEE Transactions on Audio, Speech and Language
Processing, Vol. 15, No. 8, pp. 2222–2235, 2007.
[10] K. Tokuda, T. Kobayashi, and S. Imai. Speech parameter generation from HMM using dynamic features. In Proc. ICASSP,
pp. 660–663, Detroit, U.S.A, May 1995.
[11] T. Toda, T. Muramatsu, and H. Banno. Implementation of conputationally efficient real-time voice conversion. In Proc. INTERSPEECH, Portland, Oregon, U.S., Sept. 2012.
[12] H. Zen, K. Tokuda, and T. Kitamura. Refomulating the HMM
as a trajectory model by imposing explicit relationships between static and dynamic feature vector sequences. Computer
Speech and Language, Vol. 21, No. 1, pp. 153–173, Jan. 2007.
[13] K. Tokuda, Y Nankaku, T. Toda, H. Zen, J. Yamagishi, andK. Oura. Speech synthesis based on hidden Markov models. Proceedings of the IEEE, Vol. 101, No. 5, pp. 1234–1252,
2013.
[14] T. Toda and S. Young. Trajectory training considering global
variance for HMM-based speech synthesis. In Proc. ICASSP,
pp. 4025–4028, Taipei, Taiwan, Aug. 2009.
[15] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura. Speech parameter generation algorithms for HMMbased speech synthesis. In Proc. ICASSP, pp. 1315–1318, Istanbul, Turkey, June 2000.

4863[16] H. Hwang, Y. Tsao, H. Wang, Y. Wang, and S. Chen. Incorporating global variance in the training phase of GMM-based
voice conversion. In Proc. APSIPA, pp. 1–6, Kaohsiung, Taiwan, Oct. 2013.
[17] S. Takamichi, T. Toda, G. Neubig, S. Sakti, and S. Nakamura.
A postfilter to modify modulation spectrum in HMM-based
speech synthesis. In Proc. ICASSP, pp. 290–294, Florence,
Italy, May 2014.
[18] T. Muramatsu, Y. Ohtani, T. Toda, H. Saruwatari, andK. Shikano. Low-delay voice conversion based on maximum likelihood estimation of spectral parameter trajectory.
In Proc. INTERSPEECH, pp. 1076–1079, Brisbane, Australia,
Sep. 2008.
[19] H. Zen, Y. Nankaku, and K. Tokuda. Continuous stochastic
feature mapping based on trajectory HMMs. IEEE Trans.,
Vol. 19, pp. 417–430, Jan. 2011.
[20] R. Drullman, J .M. Festen, and R. Plomp. Effect of reducing
slow temporal modulations on speech reception. J. Acoust. Soc.
of America, Vol. 95, pp. 2670–2680, 1994.
[21] S. Thomas, S. Ganapathy, and H. Hermansky. Phoneme recgnition using spectral envelop and modulation frequency features.
In Proc. ICASSP, pp. 4453–4456, Taipei, Taiwan, April 2009.
[22] J. Kominek and A. W Black. The CMU ARCTIC speech
databases for speech synthesis research. In Tech. Rep. CMULTI-03-177, Language Technologies Institute, Carnegie Mellon University, Pittsburgh, U.S.A, 2003.
[23] H. Kawahara, Jo Estill, and O. Fujimura. Aperiodicity extraction and control using mixed mode excitation and group delay
manipulation for a high quality speech analysis, modificationand synthesis system STRAIGHT . In MAVEBA 2001, pp.
1–6, Firentze, Italy, Sept. 2001.
[24] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano. Maximum
likelihood voice conversion based on GMM with STRAIGHT
mixed excitation. In Proc. INTERSPEECH, pp. 2266–2269,
Pittsburgh, U.S.A., Sept. 2006.
[25] H. Kawahara, I. Masuda-Katsuse, and A. D. Cheveigne. Restructuring speech representations using a pitch-adaptive timefrequency smoothing and an instantaneous-frequency-based
F0 extraction: Possible role of a repetitive structure in sounds.
Speech Commun., Vol. 27, No. 3–4, pp. 187–207, 1999.
[26] T. Toda, Y. Ohtani, and K. Shikano. One-to-many and many-toone voice conversion based on eigenvoices. In Proc. ICASSP,
pp. 1249–1252, Hawaii, U.S.A., Apr. 2007.
[27] Y. Ohtani, T. Toda, H. Saruwatari, and S. Shikano. Many-tomany eigenvoice conversion with reference voice. In Proc. INTERSPEECH, pp. 1623–1626, Brington U.K., Sep. 2009.