INTERSPEECH 2013

Evaluation of a Singing Voice Conversion Method Based on Many-to-Many
Eigenvoice Conversion
Hironori Doi1 , Tomoki Toda1 , Tomoyasu Nakano2 , Masataka Goto2 , Satoshi Nakamura1
1

Graduate School of Information Science,
Nara Institute of Science and Technology, Nara, Japan
2
National Institute of Advanced Industrial Science and Technology (AIST), Ibaraki, Japan1{hironori-d, tomoki, s-nakamura}@is.naist.jp, 2 {t.nakano, m.goto}@aist.go.jp

Abstract

To make it possible for people to directly sing with a different specific voice timbre, and thus overcome physical constraints, singing voice conversion has been proposed [4]. Statistical voice conversion (VC) techniques [5, 6, 7] are used to convert the singing voice timbre of a source singer into that of a target singer. In this technique, Gaussian mixture model (GMM)of the joint probability density of an acoustic feature between
the source singer’s singing voice and the target singer’s singing
voice is trained in advance using a special data set, called a parallel data set, that consists of pairs of songs of the two singers.
The trained model is capable of converting the acoustic features of the source singer’s singing voice into those of the target
singer’s singing voice for any song while keeping the linguistic information of the lyrics unchanged. Moreover, real-timesinging voice conversion can also be achieved using the lowdelay conversion algorithm [8].
Towards realizing a more flexible singing voice conversion technique, we have proposed a singing voice conversion method [9] based on many-to-many eigenvoice conversion(EVC) [10]. Many-to-many EVC is a technique of convertingfrom the voice of an arbitrary source singer into that of an arbitrary target singer. An eigenvoice GMM (EV-GMM) [11] istrained in advance using multiple parallel data sets that consist of a single predefined singer, called a reference singer inthis paper, and many prestored target singers. The EV-GMM
is capable of easily adapting the source/target voice timbre tothat of its given voice samples in a text-independent (lyricsindependent) manner. Furthermore, we have proposed a technique for efficiently and effectively generating parallel data setsusing a singing-to-singing synthesis system to artificially generate singing voices of the reference singer.
In this paper, we describe our proposed methods [9] andevaluate their effectiveness. A comparison between VC and
EVC based singing voice conversion is conducted from various perspectives. Moreover, to analyze the differences between
speaking voice and singing voice in voice conversion, we conduct comparison between singing voice conversion using EVGMM trained from speaking voice and from singing voice.

In this paper, we evaluate our proposed singing voice conversion method from various perspectives. To enable singers tofreely control their voice timbre of singing voice, we have proposed a singing voice conversion method based on many-tomany eigenvoice conversion (EVC) that enables to convert thevoice timbre of an arbitrary source singer into that of anotherarbitrary target singer using a probabilistic model. Furthermore, to easily develop training data consisting of multiple parallel data sets between a single reference singer and many othersingers, a technique for efficiently and effectively generating
the parallel data sets from nonparallel singing voice data sets ofmany singers using a singing-to-singing synthesis system havebeen proposed. However, we have never conducted sufficientinvestigations into the effectiveness of these proposed methods.
In this paper, we conduct both objective and subjective evaluations to carefully investigate the effectiveness of proposed
methods. Moreover, the differences between singing voice conversion and speaking voice conversion are also analyzed. Experimental results show that our proposed method succeeds inenabling people to control their own voice timbre by using only
an extremely small amount of the target singing voice.
Index Terms: singing voice, voice conversion, eigenvoice conversion, singing-to-singing synthesis, performance evaluation

1. Introduction

Range of singing voice timbre that can be produced by individual singers is limited by physical constraints. To produce
a singing voice beyond physical constraints, many approaches
have been studied. One of the most popular approaches isthe use of singing synthesis systems, which generate a singing
voice from several pieces of information such as lyrics andthe musical score. Among them, a text-to-singing approach,which synthesizes a singing voice from note-level score information of the melody with its lyrics, such as Vocaloid2 [1] andSinsy [2] is popular in Japan. Moreover, singing-to-singing synthesis, which automatically synthesizes a more naturally sounding singing voice by estimating the parameters of the textto-singing system from a target singing voice, has been proposed [3]. VocaListener [3], which is the system used for theestimation part of singing-to-singing synthesis, estimates parameters of pitch and dynamics for the singing synthesis system
so that the synthesized singing voice becomes more similar to
the target singing voice. If a user’s singing voice and the corresponding lyrics without any score information are available,
VocaListener can synchronize them automatically to determine
the musical note corresponding to each phoneme of the lyrics.
However, it is still difficult to generate singing voices with arbitrary and desired voice timbre.

Copyright © 2013 ISCA

2. Singing voice conversion based on
many-to-many EVC
In this section, we describe singing voice conversion method
based on many-to-many EVC and training data generation using
singing-to-singing synthesis system.
2.1. Training data generation
The development of parallel data sets consisting of singing
voice pairs of the single reference singer and many prestored1067

25 - 29 August 2013, Lyon, France

(i)mated for individual prestored target singer. Y t is the acoustic
features of the given
voiceat frame t. The 
bal source
 singer’sQ(i)ance between P w|λ(w) and Tt=1 P Y t |λ(EV ) , w is
controlled by the hyperparameter τ . The weight parameter forthe target singer ŵ(o) is estimated in the same manner. On theother hand, our proposed method allows user to freely control
voice timbre of the converted singing voice by manipulating thetarget singer’s weight parameters.
Then, the joint probability density of the acoustic features
between the source singer’s voice and the target singer’s voice
is derived
as


(i)
(o)
P Y t , Y t |ŵ(i) , ŵ(o) , λ(EV )target singers is laborious work. To address this issue, wehave artificially generated singing voices of the reference singer
by applying a singing-to-singing synthesis system to singing
voices of many prestored target singers. In this approach, weneed to prepare only singing voices of multiple prestored target
singers who need not sing the same song; these are available inexisting databases, such as the RWC Music Database [12]. For
the singing voices of each prestored target singer, corresponding
singing voices of the reference singer are artificially generated
by using the singing-to-singing synthesis system. Thus, thistraining data generation approach can efficiently and effectively
develop parallel data sets without recording singing voices of
the reference singer.
2.2. Training process
As acoustic features of the reference singer and the sth prestored target singer, we employ two D-dimensional joint fea(s)> >[x>and Y tt , ∆xt ]

=

M
X

(1)

#
"

(XY )
(XX)
Σm
Σm
µ(X)
(X,Y )m, Σm=(Y Y ) ,(Y X)
Am w(s) + bm
Σm
Σm

(2)

m=1

µ(s)
m =



To demonstrate effectiveness of our proposed method and investigate the differences between singing voice conversion andspeaking voice conversion, four types of conversion model werecompared.
VC conventional singing voice conversion based on VC [6]
EVC-human proposed singing voice conversion based on
many-to-many EVC with conventional training data generation using a human voice as the reference singer’s
voice
EVC-synth proposed singing voice conversion based on
many-to-many EVC with training data generation using
singing-to-singing synthesis

2.3. Adaptation and conversion process
In the adaptation process, the EV-GMM is adapted to an arbitrary source singer and an arbitrary target singer by independently estimating the singer-dependent weight parameterusing a few singing voice samples. The weight parameterfor source singer ŵ(i) is estimated by maximum a posteriori (MAP) [15, 16] as
T Z 

τ Y

(i)
ŵ(i) = argmax P w|λ(w)P X t , Y t |λ(EV ) , w dX t ,
w
t=1
T

τ Y


(i)
= argmax P w|λ(w)P Y t |λ(EV ) , w ,
w
t=1

λ(s)
(ω)

S
Y



P ω̂|λ(ω) ,



(i)
(i)
P Y t |X t , m, wt , λ(EV )3. Experimental evaluations

where w(s) = [w(s) (1), · · · , w(s) (J)]> is the target-speakerdependent weight parameter for controlling target voice timbre. λ(EV ) is a canonical EV-GMM parameter set consistingof the weight αm , the mean vector µ(X)m , the covariance ma(X,Y ), the bias vector bm , and the basis vectors Am =trix Σm[am (1), · · · , am (J)] for the mth mixture component, where
the number of basis vectors is J. Acoustic features of an arbitrary target speaker are modeled by setting only w(s) to the
speaker’s specific values. To alleviate the degradation of performance of EV-GMM caused by effects of acoustic variationof the many prestored target singers, the EV-GMM is trained byspeaker adaptive training (SAT) [13, 14] using multiple parallel
data sets consisting of utterance pairs of a reference and many
prestored target singers.

λ̂(ω) = argmax

Z


 

(o)
(o)
P Y t |X t , m, wt , λ(EV ) P X t |m, λ(EV ) dX t , (i)   (i) " (Y Y ) (Y XY ) #!
M
X
y
µm
Σm
Σm
=
αm N
,
(4)
(o) ;
(o)
(Y XY )(Y Y )
y
µ
Σ
Σm
m
m
m=1
where
XY )
X) (XX) −1 (XY )
Σ(Y
= Σ(Y
Σm
Σm .
(5)
m
m
In the conversion process, the converted static feature sequence
vector is estimated using the adapted EV-GMM. Maximum
likelihood estimation considering dynamic features and a global
variance [6] is adopted. Note that real-time singing voice conversion is also achieved by using the low-delay conversion algorithm [8].

(s) >
(s) >
[y t , ∆y t ]> ,(s) > >)(s),, Σ(X,Y] ; µmαm N [X >mt ,Y tP (m|λ(EV ) )m=1

tures, X t ==consisting of D-dimensional static and dynamic spectral features at frame t, respectively, where > denotes the transpositionof the vector. The joint probability density of reference and target features is modeled with the EV-GMM as follows:(s)
P (X t , Y t |λ(EV ) , w(s) )
=

M
X

(3)

s=1

where λ is a model parameter set consisting of the mean vector µ(w) and the covariance matrix Σ(ω) . This model parameter
set is trained in advance using a set of weight parameters esti-

1068

EVC-speaking conventional many-to-many EVC for a speaking voice
3.1. Experimental conditions
In this evaluation, only the spectral feature is converted in allconversion methods because the voice timbre strongly depends
on the spectral feature. The 1th to 24th mel-cepstral coefficientswere used as a spectral feature. STRAIGHT analysis [17] wasemployed to extract these coefficients from singing voices. F0
and the aperiodic components of the source singer are directly
used to synthesize the converted singing voice. The shift length
was 5 ms and the sampling frequency was 16000 Hz.
We used the solo singing voices of 30 Japanese songs in the
RWC Music Database [12] as the prestored target singing voices
to train EV-GMM. The phoneme balance was not considered inthese songs. For EVC-human, the solo singing voices of one
male singer were used as the singing voices of the reference
singer. For EVC-synth, singing voices synthesized using thesinging-to-singing synthesis system VocaListener with a singer
database called Hatsune Miku [18] based on Vocaloid2 wereMel-cepstral distortion [dB]used as the reference singer. The number of basis vectors of
the EV-GMMs was set to 29 and the number of mixture components of the EV-GMMs was set to 128. On the other hand,in EVC-speaking, we used parallel data sets of a single reference male speaker and 152 prestored target speakers to train
the EV-GMM. These speakers were from the Japanese Newspaper Article Sentence (JNAS) database. Each prestored target
speaker uttered one of seven subsets. Each subset consists of
50 phonetically balanced sentences. The EV-GMM for spectral
conversion was trained from 152 parallel data sets consisting of
the recorded reference speaking voices and the prestored target
speaking voices. The number of basis vectors of the EV-GMMs
was set to 151 and the number of mixture components of the
EV-GMMs was set to 128.
For the adaptation and testing of the EV-GMMs and for the
training and testing of the GMM, we selected two Japanesesongs from the RWC Music Database (RWC-MDB-P-2001
No.46 and No.76), which were not included in the above 30
songs. Then, 5 singers (four male singers and one female
singer) sang these two songs. Thus, as adaptation/training data
and test data, we prepared 10 songs consisting of two songs
sung by each singer. As the training data for the VC-based
method and the adaptation data for the EVC-based methods,
2, 4, 8, 16, 32, or 64% of the sung parts of songs sung by thesource and target singers was used, then, the remaining 36%
of data was used for the test. The GMM and EV-GMMs wereprepared for all combinations of the source and target singers.
Thus, for each method, 20 conversion models (10 models ×
2 song) were prepared. The weight parameters of the source
and target singer were independently estimated using the spectral features from the source and target singing voice samples.
The hyperparameter of MAP adaptation shown in eq. (3) was
preliminarily optimized in each method. In this evaluation, it
was set to 250, 1000, and 100 for EVC-human, EVC-synth, andEVC-speaking, respectively. For VC, we also trained a standard
GMM for spectral conversion using a parallel data set consisting of the source and target singing voices. The number of mixture components of the GMM was preliminarily optimized sothat the spectral conversion accuracy was maximized in the test
data.

6

VC
EVC_human
EVC_synth
EVC_speaking

5.8
5.6
5.4
5.2
5
4.8
4.6
2

4
8
16
32
64
Amount of training or adaptation data [%]Mel-cepstral distortion [dB]

Figure 1: Mel-cepstral distortion as a function of amount oftarget singing voice data (i.e., singing voice pairs in VC-based
method or singing voice adaptation data in EVC-based methods) under the same-song condition.
6

VC
EVC_human
EVC_synth
EVC_speaking

5.8
5.6
5.4
5.2
5
4.8
4.6
2

4
8
16
32
64
Amount of training or adaptation data [%]Figure 2: Mel-cepstral distortion as a function of amount oftarget singing voice data (i.e., singing voice pairs in VC-based
method or singing voice adaptation data in EVC-based methods) under the different-song condition.
EVC-based methods do not require the use of parallel data in
the adaptation, in contrast to VC.
Under the different-song condition, VC has much lower
conversion accuracy than under the same-song condition. Thisis because the voice timbre of the singing voice of a singer significantly changes depending on the song. On the other hand,
it is observed that the EVC-based methods reduce this degradation. Since the EV-GMM is trained with many singers’ voices,
it is more robust against variations of the singing voice timbre.

3.2. Objective evaluation
We evaluated two conditions of song setting: 1) the samesong condition, where the same song is used in both the training/adaptation process and the test process, and 2) the differentsong condition, where different songs are used in the training/adaptation process and the test process. Figure 1 shows melcepstral distortion as a function of the amount of the singing
voice adaptation data used in the EVC-based methods or theamount of parallel data of the singing voice pairs used in the
VC-based method under the same-song condition. Figure 2
shows those under the different-song condition. In fig. 1 and 2,
horizontal axis represents percentage of data that is used fortraining or adaptation from the sung parts of songs.
Under the same-song condition, when using a small amountof training/adaptation data, EVC-speaking is the best, EVChuman is the next, EVC-synth is the next, and VC is the worst
in conversion accuracy. Although EVC-speaking exhibits thehighest conversion accuracy, the differences from EVC-humanare not so large even if the amount of training data for EVCspeaking is significantly larger than that for EVC-human. Whenusing a large amount of training/adaptation data, VC is the best,
EVC-speaking is the next, EVC-human is the next, and the
EVC-synth is the worst in conversion accuracy. Note that the3.3. Subjective evaluation
We conducted an opinion test on the naturalness of the singing
voice and a preference test on singer individuality. The opinion was expressed using a five point scale (i.e., 1 (very poor) to
5 (excellent)). In this test, 10 listeners heard 16 types of converted singing voice sample, then they judged the naturalnessof each sample using the opinion score. In the preference test,
listeners heard a target singing voice sample and two converted
singing voice samples, then they chose the converted singing
voice sample with more similar singer individuality to the target singing voice sample. The preference test was performed
under the different-song condition because of its greater realism than same-song condition. In this tests, 9 listeners evaluatedeight types of the singing voice generated under the differentsong condition for all combinations of 2% or 64% of training/adaptation data and four types of conversion method.
Figure 3 shows the result of the opinion test on the naturalness of the singing voice. Under the same condition, VC us-

1069

Cumulative occupancy probability [%]Mean opinion score (MOS)

5
4.5
4
3.5
3
2.5
2
1.5
1
same diff same diff2

same diff same diff

64

2

VC

64

EVC_humansame diff same diff2

same diff same diff64

EVC_synth

2

64

[%]

EVC_speaking

Figure 3: Result of opinion test on naturalness.

P r efer en ce scor e [%]

40
30

64
VC

2

64

2

64

EVC-human
EVC-synth40

EVC-speaking

20
0

20

40

60

80

100

120

components of EVC-human and EVC-synth model only acoustic features of a part of prestored target speakers. Consequently,
it is expected that phonemic information and speaker individuality were not separated well in them. It is possible that thisissue causes degradation of conversion performance.
The above results suggest that 1) the proposed EVC-humanyields better conversion performance than VC when a small
amount of singing voice data of the source and target singers
is available, 2) the conversion performance of the proposed
EVC-synth is slightly degraded than EVC-human, 3) sincethese proposed methods are robust against variations of thesinging voice timbre often observed between different songs,
they work reasonably well even when different songs are usedin the adaptation and conversion processes, 4) the occupancies
of individual mixture component of EV-GMM in EVC-humanand EVC-synth are more biased than those in EVC-speaking,and then, this causes degradation of conversion accuracy forsinger individuality, 5) the differences between a speaking voice
and singing voice strongly affects to naturalness of converted
singing voice. Based on these results, to more correctly control
the voice timbre, it is necessary to train EV-GMM from larger
parallel data sets considering phoneme balance. And then, it isexpected that training data generation using singing-to-singing
synthesis is significantly effective to construct them.

50

2

60

Figure 5: Cumulative occupancy probability for all parallel
data set using several models.

95% con fiden ce in t er va ls

60

20

80

Sorted mixture component index

80
70

100

2

64

[%]

E VC_h u m a n E VC_syn t h E VC_spea kin g

Figure 4: Result of preference test on singer individuality under
the different-song condition.
ing 2% training data shows similar naturalness to that of EVChuman using 2% adaptation data in contrast to objective evaluation. On the other hand, the naturalness of EVC-speaking
is not higher than that of other methods when using a small
amount of adaptation data. This result suggests that it is difficult
for EV-GMM trained with speaking voice to generate converted
singing voice having high naturalness even if a large amount of
speaking voice is available as training data. Other results show
similar tendency to that observed in the result of the objective
evaluation.
Figure 4 shows the result of the preference test on singer
individuality. The preference score was calculated as the ratioof the number of samples selected as having better singer individuality to the number of samples presented to the listeners.
When using a small amount of training/adaptation data, EVChuman is the best, EVC-synth is the next, EVC-speaking is thenext, and VC is the worst in preference score of singer individuality. On the other hand, when using a large amount of training/adaptation data, VC and EVC-speaking show higher preference score of singer individuality than other methods. Note that
VC requires the parallel data set of the source and target singers
and the canonical EV-GMM of EVC-speaking is trained withsignificantly larger amount of training data than that of EVChuman and EVC-synth.

4. Conclusion
In this paper, we evaluated our proposed singing voice conversion methods. Our proposed methods are capable of converting
the singing voice timbre of an arbitrary source singer into thatof an arbitrary target singer by adapting a small number of adaptive parameters of a conversion model using an extremely small
amount of source and target singing voice data. Moreover, ourproposed training data generation method can alleviate the burden of having to record singing voices to develop parallel data
sets, by using a singing-to-singing synthesis system. The experimental result demonstrated that the proposed methods enable the effective conversion of a singing voice between an arbitrary singer pair even when using only several seconds of theirsinging voices as adaptation data. We plan to construct larger
parallel data sets considering phoneme balance and further improve conversion performance.

3.3.1. Comparison of each EV-GMMs
Figure 5 shows the cumulative distribution of occupancies of
the canonical EV-GMM of EVC based methods. These individual mixture component occupancies have been calculated fromall parallel data set in training process with SAT. In this figure,
we can see that the occupancies of EVC-human and EVC-synth
are more biased than that of EVC-speaking. Although the EVGMM needs to model wide varieties of acoustic features of allprestored target speakers, this result shows that some mixture

5. Acknowledgements
This work was supported in part by a JSPS KAKENHI Grant
Number 22680016 and JST OngaCREST project. The authors
are grateful to professor Hideki Kawahara of Wakayama University, Japan, for permission to use the STRAIGHT analysissynthesis method.

1070

6. References[1] H. Kenmochi and H. Ohshita, “VOCALOID – Commercial
singing synthesizer based on sample concatenation,” Proc. INTERSPEECH, pp. 4011–4012, Aug. 2007.
[2] K. Oura, A. Mase, T. Yamada, S. Muto, Y. Nankaku, andK. Tokuda, “Recent development of the HMM-based singing
voice synthesis system - Sinsy,” SSW7, pp. 211–216, Sept. 2010.
[3] T. Nakano and M. Goto, “VocaListener: A singing-to-singing synthesis system based on iterative parameter estimation,” Proc. SMC
2009, pp. 343–348, May 2009.
[4] Y. Kawakami, H. Banno, and F. Itakura, “GMM voice conversionof singing voice using vocal tract area function,” IEICE technical report. Speech 110(297) (Japanese edition), pp. 71–76, Nov.
2010.
[5] Y. Stylianou, O. Cappe, and E. Moulines, “Continuous probabilistic transform for voice conversion,” IEEE Trans. SAP, vol. 6,
no. 2, pp. 131–142, Mar. 1998.
[6] T. Toda, A. W. Black, and K. Tokuda, “Voice conversion based on
maximum likelihood estimation of spectral parameter trajectory,”
IEEE Trans. ASLP, vol. 15, no. 8, pp. 2222–2235, Nov. 2007.
[7] A. Kain and M. W. Macon, “Spectral voice conversion for text-tospeech synthesis,” Proc. ICASSP, pp. 285–288, May 1998.
[8] T. Muramatsu, Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano,“Low-delay voice conversion based on maximum likelihood estimation of spectral parameter trajectory,” Proc. INTERSPEECH,
pp. 1076–1079, Sept. 2008.
[9] H. Doi, T. Toda, T. Nakano, M. Goto, and S. Nakamura, “Singing
voice conversion method based on many-to-many eigenvoice conversion and training data generation using a singing-to-singing
synthesis system,” APSIPA ASC 2012, Dec. 2013.
[10] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Manyto-many eigenvoice conversion with reference voice,” INTERSPEECH, pp. 1623–1626, Sept. 2009.
[11] T. Toda, Y. Ohtani, and K. Shikano, “One-to-many and many-toone voice conversion based on eigenvoices,” Proc. ICASSP, pp.
1249–1252, Apr. 2007.
[12] M. Goto, T. Nishimura, H. Hashiguchi, and R. Oka, “RWC Music
Database: Music genre database and musical instrument sounddatabase,” Proc. ISMIR, pp. 229–230, Oct. 2003.
[13] T. Anastasakos, J. McDonough, S. R., and J. Makhoul, “A compact model for speaker-adaptive training,” Proc. ICSLP, vol. 2, pp.
1137–1140, 1996.
[14] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Adaptive
training for voice conversion based on eigenvoices,” IEICE Trans.
Inf. and Syst., vol. E93-D, no. 6, pp. 1589–1598, June 2010.
[15] G.-L. Gauvain and C.-H. Lee, “Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov
chains,” IEEE Trans. Speech and Audio Processing, vol. 2, no. 2,
pp. 291–298, 1994.
[16] D. Tani, T. Toda, Y. Ohtani, H. Saruwatari, and K. Shikano, “Maximum a posteriori adaptation for many-to-many eigenvoice conversion,” Proc. INTERSPEECH, pp. 1461–1464, Sept. 2008.
[17] H. Kawahara, I. Masuda-Katsuse, and A. Cheveigne, “Restructuring speech representations using a pitch-adaptive time-frequency
smoothing and an instantaneous-frequency-based f0 extraction:
Possible role of a repetitive structure in sounds,” Speech Communication, vol. 27, no. 3-4, pp. 187–207, Apr. 1999.
[18] Crypton Future Media,“What
MIKU movement”?”
2012.
http://www.crypton.co.jp/miku eng

is the ”HATSUNE[Online]. Available:

1071