IEICE TRANS. ??, VOL.Exx–??, NO.xx XXXX 200x

1

PAPER

A statistical sample-based approach to GMM-based voice
conversion using tied-covariance acoustic models
Shinnosuke TAKAMICHI†a) , Tomoki TODA††b) , Graham NEUBIG†c) , Sakriani SAKTI†d) ,and Satoshi NAKAMURA†e) ,SUMMARY
This paper presents a novel statistical sample-based approach for Gaussian Mixture Model (GMM)-based Voice Conversion (VC).
Although GMM-based VC has the promising flexibility of model adaptation, quality in converted speech is significantly worse than that of natural
speech. This paper addresses the problem of inaccurate modeling, which
is one of the main reasons causing the quality degradation. Recently, we
have proposed statistical sample-based speech synthesis using rich context
models for high-quality and flexible Hidden Markov Model (HMM)-based
Text-To-Speech (TTS) synthesis. This method makes it possible not only to
produce high-quality speech by introducing ideas from unit selection synthesis, but also to preserve flexibility of the original HMM-based TTS. In
this paper, we apply this idea to GMM-based VC. The rich context models are first trained for individual joint speech feature vectors, and then wegather them mixture by mixture to form a Rich context-GMM (R-GMM).
In conversion, an iterative generation algorithm using R-GMMs is used toconvert speech parameters, after initialization using over-trained probability distributions. Because the proposed method utilizes individual speechfeatures, and its formulation is the same as that of conventional GMMbased VC, it makes it possible to produce high-quality speech while keeping flexibility of the original GMM-based VC. The experimental results
demonstrate that the proposed method yields significant improvements interm of speech quality and speaker individuality in converted speech.
key words: GMM-based voice conversion, sample-based speech synthesis,
speech parameter conversion, rich context model

1.

Introduction

Statistical Voice Conversion (VC) is an effective techniquefor modifying speech parameters to convert non-/paralinguistic information while keeping linguistic information
unchanged, making it possible to enhance various speechbased systems [1]–[4]. While a variety of methods exist
for VC [5]–[7], Gaussian Mixture Model (GMM)-based VC[8], [9] is still popular thanks to its stability and flexibility
such as model adaptation [10]∗ .
In GMM-based VC, a GMM is used to jointly model
the source and target speech features in order to convert
the speech parameters. One of the biggest issues in GMMManuscript received January 1, 2011.
Manuscript revised January 1, 2011.
†
The author is with the Graduate School of Information Science, Nara Institute of Science and Technology, Japan††
Information Technology Center, Nagoya University, Japana) E-mail: shinnosuke-t@is.naist.jpb) E-mail: tomoki@icts.nagoya-u.ac.jpc) E-mail: neubig@is.naist.jpd) E-mail: ssakti@is.naist.jpe) E-mail: s-nakamura@is.naist.jp
DOI: 10.1587/trans.E0.??.1∗
Also, GMM-based frameworks can be combined with deepneural nets-based speech synthesis [11]–[13].

Waveform segments

Unit selection synthesis
Segment
selection
Synthetic speech

Statistical modeling

Conventional statistical approach
Parameter
generation
Synthetic speechparameters

Averaged acoustic models

Statistical modeling

Statistical sample-based approach
Selection &generation

Individual acoustic models

Synthetic speechparameters

Fig. 1 Comparison of approaches, unit selection synthesis, conventional
statistical approaches (e.g., HMM-based TTS and GMM-based VC), andthe proposed statistical sample-based approach. Whereas an acoustic
model corresponds to some speech segments in the conventional statistical approaches, it corresponds to just one speech segment in the statisticalsample-based approach. Note that the individual acoustic models are calculated using individual speech segments, but their covariance matrices are
the same to those of averaged acoustic models.

based VC is poor quality in converted speech. The reasonscausing the quality degradation are classified into analysissynthesis errors [14], the over-smoothing effect [15], [16],and inaccurate modeling, the latter of which we handle inthis paper. For example, as previous work has noted [5], [9],within each mixture component, the conversion function islinear which is a poor match for VC.
Inaccurate modeling is a common issue among statistical approaches for speech synthesis. One promising approach that has been proposed for Text-To-Speech (TTS) is
to introduce ideas of unit selection synthesis (sample-based
speech synthesis) [17], [18] which has excellent speech
quality compared to statistical approaches. For Hidden
Markov Model (HMM)-based TTS [19], Maximum Likelihood (ML)-based unit selection [20] has been proposed to
guide speech parameter segments to maximize HMM likelihoods. Although the use of individual speech segments
significantly improves quality in synthetic speech, it loses
the flexibility of the original HMM-based TTS. Recently,
we have proposed a statistical sample-based approach using

Copyright c 200x The Institute of Electronics, Information and Communication Engineers

IEICE TRANS. ??, VOL.Exx–??, NO.xx XXXX 200x

2

tied-covariance acoustic models called rich context models
for TTS that is not only high-quality but also flexible [21],as shown in Fig. 1. In conventional HMM-based TTS, because parameters of an acoustic model are estimated using
some speech segments, this approach loses information ofindividual speech segments. On the other hand, our previous work in TTS [21] has modeled the individual speechsegments with rich context models [22]. We have built the
mixture model using the rich context models, and synthetic
speech parameters are generated from the selected rich context models of the mixture model. By reformulating the statistical models in the same form as in HMM-based TTS, thismethod makes it possible to preserve the original flexibility.
In this paper, we apply this idea to GMM-based VC reviewed in Section 2. In Section 3, we construct rich context
models† corresponding to individual joint speech features,
then, construct a mixture model called a Rich context-GMM(R-GMM) using the trained rich context models belonging to
the same mixture component of the conventional GMM. An
iterative algorithm is used to convert speech parameters using the rich context models selected from the R-GMMs. For
initialization of the iteration, we further build over-trained
acoustic models to generate a less-averaged initial parameter sequence. Discontinuous transitions are observed in theinitial parameters, but it can be alleviated by the iterative
parameter conversion. The experimental results in Section
4 demonstrate that the proposed method yields significant
improvements in speech quality and speaker individuality inconverted speech.

each component has the mixture weight w(Z)q , the mean vec(Z)(Z)
tor µ(Z),andthecovariancematrixΣ.
µqqq consists of the(X)(Z)
input and output mean vectors, µq and µ(Y)q . Σq consists(XX)of the source and target covariance matrices, Σq and Σq(YY)X)
and cross-covariance matrices, Σ(Yand Σ(XY)
.
q
q
2.2

Speech parameter conversion

source speech feature sequence X =hGiven the Ti-frame>> >X1 , · · · , XT , we determine an optimal GMM mixture sequence q̂ = q̂1 , · · · , q̂t , · · · , q̂T as follows:q̂t = argmax P (q|Xt , λ) ,(4)qwhere q̂t is the optimal GMM mixture component at framet. The speech parameters are converted by maximizing an
objective function using a GMM likelihood as follows:ŷ q̂ = argmax P (Wy|X, q̂, λ)
(5)
y

−1
= W > D−1W W > D−1 E q̂ ,
(6)
q̂
q̂
h >
i>
where ŷ q̂ =ŷ1 , · · · , ŷ>t , · · · , ŷ>Tis the converted
speech parameter sequence, and ŷt is the D-dimensional
converted speech parameters at frame t.
The components of the 2DT -dimensional mean vector, E q̂ =hi>µ>q̂1 ,1 , · · · , µ>q̂t ,t , · · · , µ>q̂T ,T , and 2DT -by-2DT covariancehimatrix, D q̂ = diag2D Σq̂1 , · · · , Σq̂t , · · · , Σq̂T , are given as:2.

Conventional GMM-based voice conversionµq̂,t = Aq̂ Xt + bq̂ ,

(7)

2.1

Acoustic modelingΣq̂ = Σ(YY)− Aq̂ Σ(XX)A>q̂ ,q̂q̂

(8)

A joint probability density function of speech parametersof the source and target speakers is modeled with a GMM
using parallel data as follows:

X) (XX) −1
Aq̂ = Σ(Y
Σq̂
,
q̂
(Y)
(X)
bq̂ = µq̂ − Aq̂ µq̂ ,

(9)
(10)

w(Z)
q P (Z t |q, λ),

(1)



(Z)
,
P (Z t |q, λ) = N Z t ; µ(Z)q , Σq" (XX) (XY) #" (X) #
Σq
Σq
µq
µ(Z)
, Σ(Z)(Y X)(YY) ,
q =
q =
Σ
Σ
µ(Y)
q
q
q

(2)

where the notation diag2D denotes the construction of a
block diagonal matrix that has the 2D-by-2D diagonal elements. W is the weighting matrix to calculate the dynamic
features [23]. As shown in Eq. (7), the conventional GMM
performs linear conversion within one mixture component.

(3)

3.

Statistical sample-based voice conversion using richcontext models>
where, Z t = X>t , Y >tis the joint vector of the input spectral features Xt and the output spectral features Y t at frame t,
and Y t is given by
joint static and dynamic
 2D-dimensionalfeature vectors, y>t , ∆y>t > , where yt is represented as a Ddimensional static feature vector. The source feature vector
Xt is also given by the same form in this paper. A GMM
parameter set λ consists of the Q mixture components, and3.1

Acoustic modeling

P (Z t |λ) =

Q
X
q=1

†
The defined name “rich context model” is not strictly accurate
for GMM-based VC because no context labels are used. However,
we use this name to maintain consistency with our previous work
[21].

After conventional training, rich context models are trainedfor individual joint speech features, Z t , by updating themean vector of the GMM mixture components while tying
its covariance matrix. The m-th rich context model of theq-th GMM mixture component is(Z)
P (Z t |q, m, λ) = N Z t ; µ(Z),(11)
q,m , Σq
" (X) #
µq,m
µ(Z)
(12)
(Y) ,
q,m =
µq,m

TAKAMICHI et al.: A STATISTICAL SAMPLE-BASED APPROACH TO GMM-BASED VOICE CONVERSION USING TIED-COVARIANCE ACOUSTIC MODELS

3

where the mean vector µ(Z)q,m consists of the individual input(Y)and output mean vectors, µ(X)q,m and µq,m . The individual mean
vectors are estimated based on the ML criterion, and eachof them is equal to one joint feature vector. The mixture
component that Z t belongs to is determined as follows:q̂t = argmax P (q|Z t , λ) .

(13)

q

In this paper, we perform discriminative GMM training [24]between conventional training and rich context model training in order to alleviate mismatch between Eq. (13) and Eq.
(4)† . As described in Section 3.2, the rich context models
are selected from the mixture component determined with
Eq. (4) in speech parameter conversion. Therefore, we expect that the discriminative training is effective to select thebetter rich context models.
After training rich context models, the output probability density for each GMM mixture component is given as a
R-GMM constructed with all rich context models belonging
to the same mixture component. The R-GMM of the q-th
mixture component is
P (Z t |q, λ) =

Mq
X

w(Z)
q,m P (Z t |q, m, λ)

(14)



(Z)
(Z)
w(Z)
q,m N Z t ; µq,m , Σq ,

(15)

m=1

=

Mq
X
m=1

(Z)
where wq,mis the weight of the m-th rich context model ofthe q-th mixture component. Mq is the total number of therich context models of the q-th mixture component, and isequal to the number of speech features belonging to the mixture component. We can calculate the ML estimate of w(Z)q,mbased on the occupancy counts†† but we set it to an equiva(Z)lent value, wq,m
= 1/Mq for each component, following [21].
These procedures are shown in Fig. 2.

3.2

Speech parameter conversion

3.2.1

Iterative conversion

After determining q̂ in the standard manner with Eq.
(4), we calculate the output probability density function
P (Wy| q̂, X, λ) as follows:X
P (Wy| q̂, X, λ) =P (Wy| q̂, m, X, λ) P (m| q̂, X, λ) , (16)all mwhere m = [m1 , · · · , mt , · · · , mT ] is a rich context model
sequence, and mt is the rich context model at frame t.
P (m| q̂, X, λ) is given as:
Whereas P (q|Z t , λ) is used in the training stage, P (q|Xt , λ) isused in the conversion stage. The discriminative training algorithm[24] trains the GMM parameters to alleviate this inconsistency.
††
There are basically no duplicated joint speech features, butsuch features are included in the training data because we employ
Dynamic Time Warping (DTW) to make joint feature vectors.
†

Fig. 2 Procedure to create Rich context-GMMs (R-GMMs) using therich context models belonging to the same mixture component of the conventional GMM.

P (m| q̂, X, λ) =TYP (m|q̂t , Xt , λ) ,

(17)

t=1

P (m|q̂t , Xt , λ) ≡

1
.
Mq̂t

(18)

Traditionally, the posterior probability P (m|q̂t , Xt , λ) is calculated as the similar as Eq. (4), but we set it constantamong rich context models belonging to the same mixture
component, following [21]. In practice, there are enormous numbers of candidates for rich context models in
speech parameter conversion††† . Therefore, we calculate
P (m|q̂t , Xt , λ) in a similar fashion to Eq. (4), then, we set
P (m|q̂t , Xt , λ) = 1/Mq̂t ,t for the rich context models having
the Mq̂t ,t -best posterior probability, and P (m|q̂t , Xt , λ) = 0
otherwise, where Mq̂t ,t (1 ≤ Mq̂t ,t ≤ Mq̂t ) is the number ofcandidates at frame t.
The output probability density function is approximated with the single rich context model sequence m̂ as follows:
P (Wy| q̂, X, λ) ' P (Wy| q̂, m̂, X, λ) P ( m̂| q̂, X, λ) .

(19)

After determining the initial speech parameter sequencey(0) , the converted speech parameter sequence ŷ q̂, m̂ is deq̂, m̂
termined by iteratively maximizing the likelihood as follows:m̂(i+1) = argmax P m|W ŷ(i) , q̂, X, λ ,(20)q̂, m̂mŷ(i+1) = argmax P Wy| q̂, m̂(i+1) , X, λ ,(21)q̂, m̂
ywhere i is the iteration index. P (Wy| q̂, m̂, X, λ) is given as:
P (Wy| q̂, m̂, X, λ) = N Wy; E q̂, m̂, D q̂ ,(22)hi
E q̂, m̂ = µ>q̂1 ,m̂1 ,1 , · · · , µ>q̂T ,m̂T ,T ,(23)†††
Even if the training data size of the proposed method is thesame to that of [21], the number of the candidates becomes bigger.
This is because one rich context model corresponds to one speechfeature vector in this study whereas it corresponds to one speechsegment in [21].

IEICE TRANS. ??, VOL.Exx–??, NO.xx XXXX 200x

4

Fig. 3 Proposed initialization for iterative speech parameter conversion.
The over-trained models are trained for each sub-region divided by Eq.
(13).

µq̂,m̂,t = Aq̂ Xt + bq̂,m̂ ,

(24)

(X)
bq̂,m̂ = µ(Y)q̂,m̂ − Aq̂ µq̂,m̂ .

(25)Comparing Eq. (7) and Eq. (24), while the bias component
is constant in the conventional GMM, it varies depending on
the selected rich context models in the proposed method.
3.2.2

Initialization

For initialization of the speech parameter conversion, one
reasonable way is to use the parameter sequence generated
from conventional GMMs. However, as noted in [21], [25],it is inappropriate to initialize using the conventional GMM.
Also, we have reported in [21] that the speech parameter sequence generated from the over-trained acoustic models provides better initialization. The over-trained acoustic models efficiently avoid an averaging effect in the initialspeech parameters. Though the over-trained models cause
the discontinuous transitions in the initial parameters, thediscontinuity can be alleviated by the iterative conversion
process considering delta features. Also, we have reported
that the number of the over-trained models can be determined to maximize a Global Variance (GV) likelihood [9]of the finally generated speech parameters.
In this paper, we train the over-trained acoustic models for each sub-region as shown in Fig. 3. The acoustic
space is divided into Q sub-regions by Eq. (13) first, thenan acoustic models are trained to fit the training data of theeach sub-region. This over-trained acoustic model is given
as a GMM for each sub-region, and is trained in the standard
manner. The total number of the over-trained models is thesum of the number of mixture components of the GMMs.
The Minimum Description Length (MDL) criterion [26] canbe utilized to determine the number of over-trained models,
but we determine it by the Linde-Buzo-Gray (LBG) algorithm [27]. After determining q̂, the over-trained models are
selected as the similar as Eq. (4), and the initial parameter sequence is generated in the standard manner using theover-trained models.
3.3

Discussion

Because one rich context model corresponds to one joint

Fig. 4 An example of the conversion function within one GMM mixture
component. Whereas a conversion function of the conventional GMMbased VC is given by a linear function Aq Xt + bq , that by the rich context
models is given by a piece-wise linear function Aq Xt + bq,m , where a bias
term varies depending on individual rich context models.

feature vector, the proposed processes are related to samplebased voice conversion [28]. The target cost and concatenation cost of the sample-based approach are regarded as thelikelihoods for the static and dynamic parameters [29], [30].
From the perspective of utilizing information of individual speech features, the conversion process of the proposed method is the similar to that of kernel-based speechsynthesis [31], [32], and exemplar-based voice conversion[33], [34]. One of the comparable advantages is that the individual acoustic models can be re-selected by the iterative
conversion process.
From the perspective of the conversion function,
whereas the conventional GMM performs linear conversion
within one mixture component, the proposed method canperform piece-wise linear conversion as shown in Fig. 4.
As described in Section 3.2.2, the iterative conversion process is done to refine the discontinuity of the initial speech parameters. Fig. 5 shows the objective function
given by Eq. (19) in each iteration. Because the value isalmost converge at the 1st iteration, only a few iterations are
required.
Compared with conventional GMM-based VC, the proposed framework increases the number of model parameters
because we need (1) a conventional GMM for mixture selection, (2) over-trained models for initialization, and (3) richcontext models for conversion. As shown in Table 1, thenumber of the mean vectors notably increases as increasing the training data size. Similarly, the computation cost inconversion increases by the proposed method as shown in
Table 2, and is proportional to the training data size. On theother hand, compared with the conventional GMM-based
VC, the model complexity in the proposed method linearly
increases according to an increase of the training data size as
in unit selection synthesis. We expect that the speech quality
in the proposed method tends to be significantly improvedby increasing the training data size as in unit selection synthesis [35].

TAKAMICHI et al.: A STATISTICAL SAMPLE-BASED APPROACH TO GMM-BASED VOICE CONVERSION USING TIED-COVARIANCE ACOUSTIC MODELS

61.0443

61.030754.9452

Misclassificat ion rat e [ %]561.0447

61.0447

Mean ± standard deviation

Fig. 5 Proposed objective functions in each iteration step. “0” of the xaxis indicates the objective function just after the initialization. We can seethat the value is significantly increased at the 1st iteration, and is convergedat the 4th iteration.

4.

Experimental evaluation

4.1

Experimental condition

We selected the 450 parallel sentences of subsets A-throughI from the 503 phonetically balanced sentences included in
the ATR Japanese speech database [36] for training, and the
53 sentences of subset J for evaluation. We trained femaleto-male GMMs. Speech signals were sampled at 16 kHz.
The shift length was set to 5 ms. The 0th-through-24th
mel-cepstral coefficients were extracted as spectral parameters and log-scaled F0 and 5-band aperiodicity [37], [38]were extracted as excitation parameters. The STRAIGHT
analysis-synthesis system [39] was employed for parameter extraction and waveform generation. The feature vector consisted of spectral and excitation parameters and theirdelta and features. We built a 128-mixture GMM for spectral parameter conversion and a 16-mixture GMM for bandTable 1 The numbers of model parameters for the conventional GMMbased VC and proposed statistical sample-based VC. Mq(init) is the numberof over-trained models belonging to the q-th GMM mixture component. In
Q
QPP
our evaluation in Section 4, we set Q,
Mq(init) , and
Mq to 128, 3616,
q=1

q=1

and 590,745, respectively.
Conventional
Mixture weights

Q

ProposedQP
2Q +Mq(init)
q=1

Mean vectors

Q

Covariance matrices

Q

Q+

Q 
P
q=1

Q+

Mq(init) + Mq
Q
P

q=1



Mq(init)

Table 2 The numbers of model combinations for the conventional
GMM-based VC and proposed statistical sample-based VC. The computation cost during the model selection is proportional to these values, but thatduring the speech parameter conversion is the same between the conventional GMM-based VC, proposed initialization, and proposed conversion(for one iteration). For simplicity, the Mq̂t ,t -best approximation method
used in Section 3.2.1 is not applied here.
Conventional

Mix. selection

QT

QT

Proposed
Initialization
TP
Mq(init)tt=1

Conversion
T
P
Mqt
t=1

45
44

43.2%

43
41.8%

42
41
40

Joint

Discriminative

Fig. 6 Misclassification rate for the training data to confirm the effect ofthe discriminative GMM training.

aperiodicity conversion. The proposed method was applied
to spectral parameters. The log-scaled F0 was linearly converted. The band-aperiodicity was converted using the conventional GMM. The total number of rich context models
were 590, 745. In the parameter conversion, we selected the128-best candidates for each frame. GV [9] and modulation spectra [40] were not considered in speech parameter
conversion.
We compared the following approaches:
Conventional: conventional GMM-based VC†
Proposed: proposed approach using rich context models
Target: rich context models selected by reference data
In initialization for “Target,” the best rich context models
were selected using target reference speech parameters. We
first calculated misclassification rate for the training data to
confirm the effect of the discriminative training [24]. Then,after determining the number of over-trained models, subjective evaluations were conducted to confirm effectivenessof the proposed method.
4.2

Effect of discriminative training

We evaluated the effect of the discriminative training done
after the conventional joint density model training. The misclassification error rates were calculated for these training
algorithms. The error rate was calculated as the number ofthe misclassified training data divided by the number of the
training data. Here “misclassified data” indicates the joint
speech feature that the mixture component determined with
Eq. (13) is different from that determined with Eq. (4).
The error rates are shown in Fig. 6. Because we cansee the 1.4% reduction of the error rates, it is expected thatthe discriminative training [41] makes it possible to selectbetter rich context models.
4.3

The number of the over-trained models

We calculated GV likelihoods for the finally converted
speech parameters in order to determine the number of theover-trained models. In each sub-region, We increased thenumber with the LBG algorithm until we cannot estimate
the model parameters. Although we can change the number
sub-region by sub-region, the number was the same among†

The discriminative training [24] was performed.

IEICE TRANS. ??, VOL.Exx–??, NO.xx XXXX 200x6
1.0Preference score

TargetProposed

0.2

0.4
0.6
Com pression rat io [ %]Fig. 7 GV likelihoods for the finally converted speech parameter sequence. A compression ratio (x-axis) is the number of over-trained models
divided by that of the training data.

the sub-regions†† .
The GV likelihood is shown in Fig. 7. We can find that
the GV likelihood of “Proposed” is the biggest around thecompression ratio of 0.6 (3616 over-trained models). Therefore, we determine the number of over-trained models to be
3616.
4.4

Evaluation in speech quality and speaker individuality

In the perceptual evaluation, a preference test (AB test) was
conducted. We presented every pair of converted speech of 3
algorithms in random order, and we forced listeners to selectthe better-quality speech sample. Similarly, an XAB teston speaker individuality was conducted using the analysissynthesized speech as a reference X. 8 listeners participatedin each evaluation.
The results of the preference tests on speech quality
and speaker individuality are shown in Fig. 8 and Fig. 9,
respectively. We can find that the proposed method achieves
better scores in both speech quality and speaker individuality, compared to the conventional GMM-based VC. Therefore, we have demonstrated the effectiveness of the proposed
method. The score of “Target” is lower than that of “Proposed” in speech quality. We found some speech samples of“Target” sounds discontinuous, and it is expected that small
training data size caused this phenomenon. Whereas “Proposed” can alleviate the discontinuity by using slightly averaged initial speech parameters, “Target” uses non-averaged
initial speech parameters† . We expect that this degradation
using non-averaged parameters can be avoided by increasing the size of the training data†† . The alternative solution isto perform the iterative conversion after “Target” initialization, but this is not an aim of this study.
5.

Conclusion

This paper has proposed a novel statistical sample-based
approach using rich context models for Gaussian Mixture††
Except we cannot estimate the GMM parameters of the subregion.
†
Our previous work [21] and this study use the same size ofthe training data, but the previous work [21] did not show such a
result. We expect that this is because the rich context models usedin [21] are temporally averaged, but those of this work are not.
††The same solution is known in unit selection synthesis.

0.6
0.4
0.2
0.0

0.8

Conventional

Proposed

TargetFig. 8 Preference scores on speech quality in converted speech with 95%
confidence intervals.
1.0
Preference score

0.0

0.8

0.8
0.6
0.4
0.2
0.0

Conventional

Proposed

TargetFig. 9 Preference scores on speaker individuality in converted speechwith 95% confidence intervals.

Model (GMM)-based voice conversion. Rich context models are trained for the individual joint speech features, andthey are gathered to build a novel acoustic model called
Rich context-GMM (R-GMM). After initialization using theover-trained acoustic models, speech parameters are iteratively converted using the selected rich context models. Experimental evaluation has demonstrated that the proposed
method achieves better scores in both speech quality andspeaker individuality. we will further integrate the GV andmodulation spectra modeling techniques to the rich context
models, and also investigate an adaptation method using richcontext models.
Acknowledgements: Part of this work was supported
by Grant-in-Aid for JSPS Fellows Grant Number 26·10354.
References[1] K. Tanaka, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, “A hybrid
approach to electrolaryngeal speech enhancement based on noise reduction and statistical excitation generation,” IEICE Transactions on
Information and Systems, vol. E97-D, no. 6, pp. 1429–1437, Jun.
2014.
[2] K. Kobayashi, T. Toda, H. Doi, T. Nakano, M. Goto, G. Neubig,
S. Sakti, and S. Nakamura, “Voice timbre control based on perceived
age in singing voice conversion,” IEICE Transactions on Information and Systems, vol. E97-D, no. 6, pp. 1419–1428, Jun. 2014.
[3] N. Hattori, T. Toda, H. Kawai, H. Saruwatari, and K. Shikano,“Speaker-adaptive speech synthesis based on Eigenvoice conversion
and language-dependent prosodic conversion in speech-to-speechtranslation,” in Proc. INTERSPEECH, Florence, Italy, Aug. 2011,
pp. 2769–2772.
[4] S. Aryal and R. G.-Osuna, “Can voice conversion be used to reduce
non-native accents?” in Proc. ICASSP, Florence, Italy, May 2014,
pp. 7929–7933.
[5] T. Nakashika, R. Takashima, T. Takiguchi, and Y. Ariki, “Voice conversion in high-order Eigen space using deep belief nets,” in Proc.
INTERSPEECH, Lyon, France, Aug. 2013, pp. 369–372.
[6] Z. Wu, T. Virtanen, T. Kinnunen, E. S. Chng, and H. Li, “Exemplar-

TAKAMICHI et al.: A STATISTICAL SAMPLE-BASED APPROACH TO GMM-BASED VOICE CONVERSION USING TIED-COVARIANCE ACOUSTIC MODELS

7

based voice conversion using non-negative spectrogram deconvolution,” in Proc. SSW8, Catalunya, Spain, Aug. 2013.
[7] E. Helander, H. Silén, T. Virtanen, and M. Gabbouj, “Voice conversion using dynamic kernel partial least squares regression,” IEEE
Transactions on Audio, Speech, and Language Processing, vol. 20,
no. 3, pp. 806–817, Mar. 2012.
[8] Y. Stylianou, O. Cappé, and E. Moulines, “Continuous probabilistictransform for voice conversion,” IEEE Transactions on Speech andAudio Processing, vol. 6, no. 2, pp. 131–142, Mar. 1988.
[9] T. Toda, A. W. Black, and K. Tokuda, “Voice conversion based on
maximum likelihood estimation of spectral parameter trajectory,”
IEEE Transactions on Audio, Speech, and Language Processing,vol. 15, no. 8, pp. 2222–2235, 2007.
[10] K. Ohta, T. Toda, Y. Ohtani, H. Saruwatari, and K. Shikano, “Adaptive voice-quality control based on one-to-many Eigenvoice conversion,” in Proc. INTERSPEECH, Chiba, Japan, Sep. 2010, pp. 2158–
2161.
[11] L.-H. Chen, Z.-H. Ling, Y. Song, and L.-R. Dai, “Joint spectral distribution modeling using restricted Boltzmann machines for voice
conversion,” in Proc. INTERSPEECH, Lyon, France, Sep. 2013, pp.
3052–3056.
[12] Z. Wu, P. Swietojanski, C. Veaux, S. Renals, and S. King, “A studyof speaker adaptation for DNN-based speech synthesis,” in Proc.
INTERSPEECH, Dresden, Germany, Sep. 2015, pp. 879–883.
[13] E. Variani, E. McDermott, and G. Heigold, “A Gaussian mixture
model layer jointly optimized with discriminative features within a
deep neural network architecture,” in Proc. ICASSP, Brisbane, Australia, Apr. 2015.
[14] K. Kobayashi, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, “Statistical singing voice conversion with direct waveform modification
based on the spectrum differential,” in Proc. INTERSPEECH, Max
Atria, Singapore, Sep. 2014, pp. 2514–2518.
[15] S. Takamichi, T. Toda, A. W. Black, G. Neubig, S. Sakti, and
S. Nakamura, “Postfilters to modify the modulation spectrum forstatistical parametric speech synthesis,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 24, no. 4, pp. 755–767,
2016.
[16] S. Takamichi, T. Toda, A. W. Black, and S. Nakamura, “Modulation
spectrum-constrained trajectory training algorithm for GMM-based
voice conversion,” in Proc. ICASSP, Brisbane, Australia, Apr. 2015.
[17] N. Iwahashi, N. Kaiki, and Y. Sagisaka, “Speech segment selection
for concatenative synthesis based on spectral distortion minimization,” IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, vol. E76-A, no. 11, pp. 1942–
1948, 1993.
[18] A. J. Hunt and A. Black, “Unit selection in a concatenative speechsynthesis system using a large speech database,” in Proc. ICASSP,
Atlanta, U.S.A., May 1996, pp. 373–376.
[19] K. Tokuda, Y. Nankaku, T. Toda, H. Zen, J. Yamagishi, and K. Oura,“Speech synthesis based on hidden Markov models,” Proceedings ofthe IEEE, vol. 101, no. 5, pp. 1234–1252, 2013.
[20] Z. Ling, L. Qin, H. Lu, Y. Gao, L. Dai, R. Wang, Y. Jiang, Z. Zhao,
J. Yang, J. Chen, and G. Hu, “The USTC and iflytek speech synthesis
systems for blizzard challenge 2007,” in Proc. Blizzard Challenge
workshop, Bonn, Germany, Aug. 2007.
[21] S. Takamichi, T. Toda, Y. Shiga, S. Sakti, G. Neubig, and S. Nakamura, “Parameter generation methods with rich context models forhigh-quality and flexible text-to-speech synthesis,” IEEE Journal of
Selected Topics in Signal Processing, vol. 8, no. 2, pp. 239–250,
2014.
[22] Z. Yan, Q. Yao, and S. K. Frank, “Rich context modeling forhigh quality HMM-based TTS,” in Proc. INTERSPEECH, Brighton,
U.K., Sep. 2009, pp. 1755–1758.
[23] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura, “Speech parameter generation algorithms for HMM-based
speech synthesis,” in Proc. ICASSP, Istanbul, Turkey, June 2000,
pp. 1315–1318.

[24] H.-T. Hwang, Y. Tsao, H.-M. Wang, Y.-R. Wang, and S.-H. Chen,“Alleviating the over-smoothing problem in GMM-based voice conversion with discriminative training,” in Proc. INTERSPEECH,
Lyon, France, Sep. 2013, pp. 3062–3066.
[25] T. Merritt, J. Yamagishi, Z. Wu, O. Watts, and S. King, “Deep neural network context embeddings for model selection in rich-context
HMM synthesis,” in Proc. INTERSPEECH, Dresden, Germany, Sep.
2015, pp. 2207–2211.
[26] K. Shinoda and T. Watanabe, “MDL-based context-dependent subword modeling for speech recognition,” Journal of the AcousticalSociety of Japan (E), vol. 28, no. 3, pp. 140–146, 2007.
[27] Y. Linde, A. Buzo, and R. M. Gray, “An algorithm for vector quantizer design,” IEEE Transactions on Communications, pp. 84–95,
1980.
[28] T. Dutoit, A. Holzapfel, M. Jottrand, A. Moinet, J. Perez, and
Y. Stylianou, “Towards a voice conversion system based on frame
selection,” in Proc. ICASSP, Hawaii, U.S.A., Apr. 2007, pp. 513–
516.
[29] S. Kataoka, N. Mizutani, K. Tokuda, and T. Kitamura, “Decision
tree backing-off in HMM-based speech synthesis,” in Proc. INTERSPEECH, vol. 2, Jeju, Korea, Oct. 2004, pp. 1205–1208.
[30] Z. Ling and R. Wang, “HMM-based unit selection using frame sized
speech segments,” in Proc. INTERSPEECH, Pittsburgh U.S.A., Sep.
2006, pp. 2034–2037.
[31] T. Koriyama, T. Nose, and T. Kobayashi, “Statistical parametricspeech synthesis based on Gaussian process regression,” IEEE Journal of Selected Topics in Signal Processing, vol. 8, no. 2, pp. 173–
183, Apr. 2014.
[32] H. Z. N. C. V. Pilkington and M. J. F. Gales, “Gaussian processexperts for voice conversion,” in Proc. INTERSPEECH, Florence,
Italy, Jul. 2011.
[33] R. Aihara, T. Takiguchi, and Y. Ariki, “Activity-mapping nonnegative matrix factorization for exemplar-based voice conversion,”in Proc. ICASSP, Brisbane, Australia, Apr. 2015, pp. 4899–4903.
[34] Z. Wu, E. S. Chng, and H. Li, “Exemplar-based voice conversion
using joint nonnegative matrix factorization,” Multimedia Tools andApplications, vol. 74, no. 22, pp. 9943–9958, 2015.
[35] T. Toda, H. Kawai, M. Tsuzaki, and K. Shikano, “An evaluation of
cost functions sensitively capturing local degradation of naturalness
for segment selection in concatenative speech synthesis,” SpeechCommunication, vol. 48, no. 1, pp. 45–56, 2006.
[36] Y. Sagisaka, K. Takeda, M. Abe, S. Katagiri, T. Umeda, andH. Kuawhara, “A large-scale Japanese speech database,” in ICSLP90, Kobe, Japan, Nov. 1990, pp. 1089–1092.
[37] H. Kawahara, J. Estill, and O. Fujimura, “Aperiodicity extraction
and control using mixed mode excitation and group delay manipulation for a high quality speech analysis, modification and synthesis
system STRAIGHT,” in MAVEBA 2001, Firentze, Italy, Sep. 2001,
pp. 1–6.
[38] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Maximum likelihood voice conversion based on GMM with STRAIGHT mixed
excitation,” in Proc. INTERSPEECH, Pittsburgh, U.S.A., Sep. 2006,
pp. 2266–2269.
[39] H. Kawahara, I. Masuda-Katsuse, and A. D. Cheveigne, “Restructuring speech representations using a pitch-adaptive time-frequency
smoothing and an instantaneous-frequency-based F0 extraction:
Possible role of a repetitive structure in sounds,” Speech Communication, vol. 27, no. 3–4, pp. 187–207, 1999.
[40] S. Takamichi, T. Toda, A. W. Black, and S. Nakamura, “Parameter
generation algorithm considering modulation spectrum for HMMbased speech synthesis,” in Proc. ICASSP, Brisbane, Australia, Apr.
2015, pp. 4210–4214.
[41] H. Hwang, Y. Tsao, H. Wang, Y. Wang, and S. Chen, “Incorporating
global variance in the training phase of GMM-based voice conversion,” in Proc. APSIPA ASC, Kaohsiung, Taiwan, Oct. 2013, pp. 1–6.

IEICE TRANS. ??, VOL.Exx–??, NO.xx XXXX 200x8

Shinnosuke TAKAMICHI
received his
B.E. from Nagaoka University of Technology,
Japan, in 2011 and his M.E. and D.E. degrees
from the Graduate School of Information Science, Nara Institute of Science and Technology(NAIST), Japan, in 2013 and 2016, respectively.
He was a short-time researcher at the NICT,
Kyoto, Japan in 2013, a visiting researcher ofCarnegie Mellon University (CMU) in UnitedStates, from 2014 to 2015, and Research Fellow (DC2) of Japan Society for the Promotionof Science, from 2014 to 2016. He is currently a Project Research Associate of the University of Tokyo. He received the 7th Student Presentation
Award from ASJ, the 35th Awaya Prize Young Researcher Award from
ASJ, the 8th Outstanding Student Paper Award from IEEE Japan Chapter
SPS, the Best Paper Award from APSIPA ASC 2014, the Student Paper
Award from IEEE Kansai Section, the 30th TELECOM System Technology Award from TAF, the 2014 ISS Young Researcher’s Award in Speech
Field from the IEICE, the NAIST Best Student Award (Ph.D course), andthe Best Student Award of Graduate School of Information Science (Ph.D
course). His research interests include electroacoustics, signal processing,and speech synthesis. He is a student member of ASJ and IEEE SPS, and amember of ISCA.

Tomoki TODA
earned his B.E. degree from
Nagoya University, Aichi, Japan, in 1999 andhis M.E. and D.E. degrees from the Graduate
School of Information Science, NAIST, Nara,
Japan, in 2001 and 2003, respectively. He is a
Professor at the Information Technology Center,
Nagoya University. He was a Research Fellowof JSPS from 2003 to 2005. He was then an Assistant Professor (2005-2011) and an Associate
Professor (2011-2015) at the Graduate School ofInformation Science, NAIST. His research interests include statistical approaches to speech processing. He received morethan 10 paper/achievement awards including the IEEE SPS 2009 Young
Author Best Paper Award and the 2013 EURASIP-ISCA Best Paper Award(Speech Communication Journal).

Graham NEUBIG
received his B.E. from
University of Illinois, Urbana-Champaign in
2005, and his M.S. and Ph.D. in informatics
from Kyoto University in 2010 and 2012 respectively. From 2012, he has been an assistant professor at the Nara Institute of Science and Technology, where he is pursuing research in machine translation and spoken language processing.

Sakriani SAKTI
received her B.E. degree
in Informatics (cum laude) from Bandung Institute of Technology, Indonesia, in 1999. In 2000,
she received DAAD-Siemens Program Asia 21stCentury Award to study in Communication
Technology, University of Ulm, Germany, andreceived her MSc degree in 2002. During
her thesis work, she worked with Speech Understanding Department, DaimlerChrysler Research Center, Ulm, Germany. Between 20032009, she worked as a researcher at ATR SLC

Labs, Japan, and during 2006-2011, she worked as an expert researcher
at NICT SLC Groups, Japan. While working with ATR-NICT, Japan, she
continued her study (2005-2008) with Dialog Systems Group University
of Ulm, Germany, and received her PhD degree in 2008. She actively
involved in collaboration activities such as Asian Pacific Telecommunity
Project (2003-2007), A-STAR and U-STAR (2006-2011). In 2009-2011,
she served as a visiting professor of Computer Science Department, University of Indonesia (UI), Indonesia. From 2011, she has been an assistantprofessor at the Augmented Human Communication Laboratory, NAIST,Japan. She served also as a visiting scientific researcher of INRIA ParisRocquencourt, France, in 2015-2016, under ”JSPS Strategic Young Researcher Overseas Visits Program for Accelerating Brain Circulation”. She
is a member of JNS, SFN, ASJ, ISCA, IEICE and IEEE. Her research interests include statistical pattern recognition, speech recognition, spoken
language translation, cognitive communication, and graphical modelingframework.

Satoshi NAKAMURA
is Professor of Graduate School of Information Science, Nara Institute of Science and Technology, Japan, Honorarprofessor of Karlsruhe Institute of Technology, Germany, and ATR Fellow. He received his
B.S. from Kyoto Institute of Technology in 1981
and Ph.D. from Kyoto University in 1992. He
was Associate Professor of Graduate School ofInformation Science at Nara Institute of Scienceand Technology in 1994-2000. He was Directorof ATR Spoken Language Communication Research Laboratories in 2000-2008 and Vice president of ATR in 2007-2008.
He was Director General of Keihanna Research Laboratories and the Executive Director of Knowledge Creating Communication Research Center,
National Institute of Information and Communications Technology, Japan
in 2009-2010. He is currently Director of Augmented Human Communication laboratory and a full professor of Graduate School of Information Science at Nara Institute of Science and Technology. He is interested in modeling and systems of speech-to-speech translation and speech recognition.
He is one of the leaders of speech-to-speech translation research and has
been serving for various speech-to-speech translation research projects inthe world including C-STAR, IWSLT and A-STAR. He received Yamashita
Research Award, Kiyasu Award from the Information Processing Societyof Japan, Telecom System Award, AAMT Nagao Award, Docomo Mobile
Science Award in 2007, ASJ Award for Distinguished Achievements in
Acoustics. He received the Commendation for Science and Technology by
the Minister of Education, Science and Technology, and the Commendationfor Science and Technology by the Minister of Internal Affair and Communications. He also received LREC Antonio Zampoli Award 2012. He has
been Elected Board Member of International Speech Communication Association, ISCA, since June 2011, IEEE Signal Processing Magazine Editorial Board Member since April 2012, IEEE SPS Speech and Language
Technical Committee Member since 2013, and IEEE Fellow since 2016.