INTERSPEECH 2015

Statistical Singing Voice Conversion based on Direct Waveform Modification
with Global Variance
Kazuhiro Kobayashi, Tomoki Toda, Graham Neubig, Sakriani Sakti, Satoshi Nakamura
Graduate School of Information Science, Nara Institute of Science and Technology (NAIST), Japan{kazuhiro-k, tomoki, neubig, ssakti, s-nakamura}@is.naist.jp

Abstract

lyrics on the fly.
Singing voice conversion (SVC), on the other hand, converts a source singer’s singing voice into another target singer’s
singing voice [6, 7]. This makes it possible to produce the desired singing voices on the fly, enabling singers to sing songs
with their desired voice timbre, not limited by physical constraints. One of the typical methods is based on statisticalvoice conversion (VC) techniques [8, 9]. A conversion model istrained in advance using acoustic features, which are extracted
from a parallel data set of song pairs sung by the source and target singers. The trained conversion model makes it possible toconvert the acoustic features of the source singer’s singing voice
into those of the target singer’s singing voice in any song while
keeping the linguistic information of the lyrics unchanged. Recently eigenvoice conversion (EVC) techniques [10, 11] havealso been successfully applied to SVC [12] to develop more
flexible SVC systems capable of achieving conversion between
arbitrary source and target singers, even if a parallel data set
is not available. However, speech quality of the singing voice
converted by SVC is usually degraded compared to that of thenatural singing voice due to various errors caused by not only
the acoustic feature conversion process, but also the vocoding
process for waveform generation.
To improve speech quality of the converted singing voice,
we have proposed an SVC method with direct waveform
modification based on spectrum differential (DIFFSVC) [13].
DIFFSVC can avoid using the vocoder framework in generation of the excitation signal by directly filtering an input singing
voice waveform with a time sequence of spectral feature differentials estimated by a differential Gaussian mixture model(GMM) derived from the conventional GMM used in the standard SVC method. Although DIFFSVC is applicable to only
situations in which pitch conversion is not necessary (such as inintra gender conversion), voice timbre of the input singing voice
can be successfully converted into that of the target singer whileachieving speech quality significantly higher than the standard
SVC method. The direct waveform filtering tends to keep modulation components of the converted spectral parameter trajectory larger compared to those in the standard SVC with thevocoder-based waveform generation. However, they are still
significantly smaller than those of natural spectral parameter
trajectories as the converted spectral parameter trajectory tends
to be excessively smoothed. This over-smoothing effect is well–known as a factor causing quality degradation in the synthesized
sining voice.
In this paper, to alleviate the over-smoothing effect, we propose a parameter generation algorithm considering global variance (GV) for DIFFSVC. GV is a well-known feature to measure the over-smoothing effect [9]. To restore the GV of theconverted spectral parameter trajectory, we modify the objective

This paper presents techniques to improve the quality of voices
generated through statistical singing voice conversion withdirect waveform modification based on spectrum differential(DIFFSVC). The DIFFSVC method makes it possible to convert singing voice characteristics of a source singer into thoseof a target singer without using vocoder-based waveform generation. However, quality of the converted singing voice stilldegrades compared to that of a natural singing voice due to various factors, such as the over-smoothing of the converted spectral parameter trajectory. To alleviate this over-smoothing, we
propose a technique to restore the global variance of the converted spectral parameter trajectory within the framework of
the DIFFSVC method. We also propose another technique tospecifically avoid over-smoothing at unvoiced frames. Resultsof subjective and objective evaluations demonstrate that the proposed techniques significantly improve speech quality of theconverted singing voice while preserving the conversion accuracy of singer identity compared to the conventional DIFFSVC.
Index Terms: statistical singing voice conversion, direct waveform modification, spectral differential, global variance, Gaussian mixture model

1. Introduction
A singing voice is one of the most expressive components inmusic. In addition to pitch, dynamics, and rhythm, the linguistic information of the lyrics can be used by singers to express
more varieties of expression than other music instruments. Although singers can also expressively control their voice timbre
to some degree, they usually have a difficulty in changing it
widely (e.g. changing their own voice timbre into that of another specific singer) owing to physical constraints in speech
production. If singers could freely control their voice timbre
beyond their physical constraints, it would open up entirely new
ways for singers to express more varieties of expression.
Singing synthesis [1, 2, 3] has been a growing interest incomputer-based music technology. Entering notes and lyrics tothe singing synthesis engine, users (e.g., composers and singers)
can easily produce a synthesized singing voice which has a specific singer’s voice characteristics, different from those of theusers. Previous work has proposed techniques to flexibly control the synthesized singing voice as the users want by automatically adjusting parameters of the singing synthesis engine sothat the variation of power and pitch in the synthesized singing
voice is similar to that of the given users’ natural singing voice[4, 5]. Although these technologies using singing synthesis engines are effective to produce the singing voices desired by theusers, it is essentially difficult to produce synthesized singing
voices by controlling all singing voice components including

Copyright © 2015 ISCA

2754

September 6- 10, 2015, Dresden, Germany

function to determine a time sequence of the spectral differential. Additionally, we implement a process smoothing the spectral differential at unvoiced frames to avoid the over-smoothing
effect at unvoiced sounds. We conduct subjective and objective
evaluations, demonstrating that the proposed DIFFSVC method
significantly improves speech quality of the converted singing
voice compared to the conventional DIFFSVC method.

the source features and the spectrum feature differential are de⊤ ⊤⊤ ⊤noted as X = [X ⊤and D = [D ⊤1 , · · · , XT ]1 , · · · , DT ]where T is the number of frames included in the time sequenceof the given source feature vectors. A time sequence vector of⊤⊤
the converted static features d̂ = [d̂1 , · · · , d̂T ]⊤ is determined
as follows:d̂ = argmax P (D|X, λ) s.t. D = W d
(6)
d
T !
M
*
P (D|X, λ) =P (m|X t , λ)P (D t |m, X t , λ) (7)2. Statistical singing voice conversion withdirect waveform modification (DIFFSVC)DIFFSVC consists of a training process and a conversion process. In the training process, a joint probability density function of spectral features of a source singer and the differential
between the source and target singers is modeled with a differential GMM, which is directly derived from a traditional GMM.
As the spectral features of the source and target singers, we employ 2D-dimensional joint static and dynamic feature vectors⊤ ⊤⊤ ⊤X t = [x⊤of the source and Y t = [y ⊤oft , ∆xt ]t , ∆y t ]the target consisting of D-dimensional static feature vectors xt
and y t and their dynamic feature vectors ∆xt and ∆y t at framet, respectively, where ⊤ denotes the transposition of the vector.
As shown in [7], their joint probability density modeled by the
GMM is given by

t=1 m=1

where W is a transformation matrix to expand the static feature
vector sequence into the joint static and dynamic feature vector
sequence [14] and the probability density function at frame t isgiven by+,
(D)
P (D t |m, X t , λ) = N D t ; E m,t , V (D)
(8)
m
(D)

(DX) (XX)E m,t = µ(D)Σm
m + Σm

V (D)
m

"# $#
$% (XX) (XY ) &'X t µ(X)
Σm Σm
m
=
αm N
;
) ,
(Y X)
(Y Y )
Y t µ(Y
Σm Σm
m
m=1

P (X t , D t |λ)"# $#$% (XX) (XD) &'M!
X t µ(X)
Σm Σm
=
αm N
; m
,
(DX)
(DD)
D t µ(D)Σm Σmm−1 (XD)
Σ(DD)− Σ(DX)
Σ(XX)
Σm .
m
m
m

(2)

)
(X)
µ(D)
= µ(Y
m
m − µm(3)v(y) = [v(1), v(2), · · · , v(d), · · · , v(D)]⊤

,

(9)

(10)

T
!

v(d) =

1(yt (d) − ȳ(d))2
T t=1

ȳ(d) =1 !
yτ (d)T τ =1

(11)
(12)

T

(13)

where yt (d) shows the d-th component of the target feature vector at frame t. The probability density function of the GV ismodeled as follows:+,+,
P v(y) | λ(v) = N v(y) ; µ(v) , Σ(vv)
(14)

m=1

Σ(XD)
=
m
Σ(DD)
=
m

X t − µ(X)m

In order to improve the speech quality of the converted singing
voice in DIFFSVC, we propose two techniques: 1) restorationof the GV of the converted spectral feature trajectory and 2)
smoothing of the converted spectral feature differential at unvoiced frames.
The GV of the target static feature vector over the time sequence is written as(1)

where N (·; µ, Σ) denotes a Gaussian distribution with a mean
vector µ and a covariance matrix Σ. The mixture component index is m. The total number of mixture components
is M . λ is a GMM parameter set consisting of the mixturecomponent weight αm , the mean vector µm , and the covariance matrix Σm of the m-th mixture component. The GMM
is trained using joint vectors of X t and Y t in the parallel dataset, which are automatically aligned to each other by dynamic
time warping. Then, the differential GMM is analytically derived from the trained GMM by transforming the parameters.
()⊤ ⊤Let D t = d⊤denote the static and dynamic differt , ∆dt
ential feature vector, where dt = y t − xt . The joint probability
density function of the source and differential spectral features
is shown as follows:

⊤
)
Σ(DX)
= Σ(XY− Σ(XX)mm
m(XX)(Y Y ))X)Σm+ Σm− Σ(XY
− Σ(Y
.
m
m

+

3. DIFFSVC considering global variance

P (X t , Y t |λ)
M
!

=

−1

where λ(v) is a parameter set of a Gaussian distribution forwhich the mean vector and covariance matrix are µ(v) andΣ(vv) , respectively. The converted feature differential trajectory is determined by maximizing a new objective function as
follows:(4)(5)

d̂ = argmax P (D|X, λ)ω P (v(y ′ )|λ(v) ) s.t. D = W d (15)d

In the conversion process, the converted spectral feature differential is estimated from the source singer’s spectral features
based on the differential GMM in the same manner as maximum
likelihood estimation of speech parameter trajectory with the
GMM [9]. The voice timbre of the source singer is converted
into that of the target singer by directly filtering the speech
waveform of the input natural singing voice with the converted spectral feature differential. Time sequence vectors of

where y ′ = [x + d] and the constant ω denotes a parameterfor controlling the balance between the two likelihoods. The
converted feature differential trajectory is iteratively updated byusing the steepest descent method as follows:
d̂

2755

(i+1)−th= d̂(i)−th+ α · ∆d̂(i)−th(16)

(i)−th100

Preference score [%]where α is a step size parameter. The gradient vector ∆d̂
is given by∂L (i)−th∆d=(17)∂d d=d(i)−th+,−1−1 (D)∂L= ω −W ⊤ V (D)W d + W ⊤ V (D)
Em
m
m
∂d
.
/
⊤
⊤
⊤
⊤ ⊤
+ v ′1 , v ′2 , · · · , v ′t , · · · , v ′T

E (D)
m =
V (D)
m

−1

.

(D)
(D)
(D)
E m1 ,1 , · · · , E mt ,t , · · · , E mT ,T.

= diag V (D)m1−1, · · · , V (D)
mt

−1

/⊤

, · · · , V (D)
mT

60
40
20
0

(18)

(a)

(b)

Figure 1: Results of preference test. (a) speech quality of converted singing voice, (b) conversion accuracy of singer individuality.

(19)
/
−1

(
)⊤
v ′t = vt′ (1), vt′ (2), · · · , vt′ (d), · · · , vt′ (D)80Conventional DIFFSVC
Proposed DIFFSVC
95% confidence interval(20)

duration of each phrase was approximately 8 seconds. 3 males
and 3 females sang these phrases. The sampling frequency was
set to 16 kHz.
STRAIGHT [16] was used to extract spectral envelopes,
which were parameterized to the 1-24th mel-cepstral coefficients as the spectral features. The frame shift was 5 ms. The
mel log spectrum approximation (MLSA) filter [17] was usedas the synthesis filter.
We used 80 randomly selected phrases for the GMM training and the remaining 72 phrases were used for evaluation.
The speaker-dependent GMMs were separately trained for individual singer pairs determined in a round-robin fashion within
intra-gender singers. The number of mixture components was
128.
Two preference tests were conducted. The first test evaluated speech quality of the converted singing voices. The converted singing voice samples of the conventional and proposed
DIFFSVC methods for the same phrase were presented to listeners in random order. The listeners selected which sample
had better sound quality. The second preference test evaluatedthe singer identity conversion accuracy. A natural singing voicesample of the target singer was presented to the listeners firstas a reference. Then, the converted singing voice samples of
the conventional and proposed DIFFSVC methods for the samephrase were presented in random order. The listeners selectedwhich sample was more similar to the reference natural singing
voice in terms of singer identity. The number of listeners was 6
and each listener evaluated 54 sample pairs. They were allowed
to replay each sample pair as many times as necessary.

(21)
+
,0
1
⊤
2
vt′ (d) = − p(v) (d) v(y ′ ) − µ(v) yt′ (d) − y¯′ (d)T(22)
where p(v) (d) indicates the d-th column vector of the inverse
matrix of Σ(vv) . An initial feature differential trajectory for theiterative update is determined by filtering in the conventional
DIFFSVC as follows:2µv (d)′dˆt (d) =(ŷt (d) − ŷ(d)) + ŷ(d) − xt (d)
(23)
v(d)
where ŷt (d) indicates the converted feature at frame t determined by the conventional DIFFSVC and ŷ(d) indicates its average over a time sequence.
It has been reported that unvoiced consonants (e.g. /s/, /sh/)are less affected by speaker individuality compared to voiced
sounds (e.g. /ae/, /n/) in normal speech [15]. Based on thisfinding, in order to alleviate the over-smoothing effect as muchas possible, we minimize the amount of conversion at unvoiced
frames by smoothing the converted feature differential at thoseframes. We implement this process on top of the previously(D)(D) −1
described DIFFSVC with GV by modifying E m,t and V mat unvoiced frames as follows:3(D)
0 (for static & delta)
E m,t =
(24)
4
0(for static)
−1
V (D)
=
(25)
m
(∆D) −1Vm(for delta)

4.2. Subjective evaluation
Figure 1 (a) indicates the result of the preference test for the
speech quality. The proposed DIFFSVC method generates theconverted speech with better speech quality than the conventional DIFFSVC method. Figure 1 (b) indicates the result of thepreference test for the singer identity. The conversion accuracyof the singer identity of the proposed DIFFSVC method is not
significantly different from that of the conventional DIFFSVC
method. Although the proposed DIFFSVC method avoids accurately converting spectral features at unvoiced frames, it stillyields conversion accuracy of singer individuality almost equal
to that of the conventional DIFFSVC method.
These results demonstrate that the proposed DIFFSVC
method is capable of converting voice timbre with higher
speech quality while causing no degradation in the conversion accuracy of singer identity compared to the conventional
DIFFSVC method.

(∆D) −1
where V mshows delta components of the inverse matrixof the covariance matrix in Eq. (8). These parameter modifications make the converted spectral feature differential smoothly
vary at unvoiced frames. Note that we avoid updating the converted spectral feature differential at the unvoiced frames in Eq.
(16).

4. Experimental evaluation
4.1. Experimental conditions
We evaluated speech quality and singer identity of the converted singing voices to compare the conventional and proposed
DIFFSVC methods. We used singing voices of 21 Japanese traditional songs, which were divided into 152 phrases, where the2756Log F0
Mel-cepstrum coefficient

7
6
5
4
3
2
1
0
0
4
3
2
1
0
-1
-2
-3
0
1.0
0.5
0.0
-0.5
-1.0
-1.5
-2.0
0
0.4
0.2
0.0
-0.2
-0.4
-0.6
-0.8
0

100

200

100

200

100

200

100

200

300

400

300

400

500

300

400

500

300

400

500

1-st dimension

4-th dimension

16-th dimension

7
6
5
4
3
2
1
0
0
4
3
2
1
0
-1
-2
0
600
1.0
0.5
0.0
-0.5
-1.0
-1.5
600
0
0.2
0.0
-0.2
-0.4
-0.6
600
0

w/o GV (filtered)w/ GV (filtered)

SourceTargetw/o GV (estimated)
w/ GV
500(estimated) 600

100

200

100

200

100

200

100

200

Frame index

300

400

500

600

300

400

500

600

300

400

500

600

300

400

500

600

1-st dimension

4-th dimension

16-th dimension

Frame index

Figure 2: Example of trajectories of spectral feature sequences. Note that the duration of “Target” trajectories is different from theother trajectories.
10

4.3. Analysis of converted feature trajectories

Global variance

To more deeply analyze what yields naturalness improvementsin the proposed DIFFSVC method, we examine in detail the
spectral feature trajectories of singing voices, which are given
bySource mel-cepstral coefficients extracted from the source
singer’s natural singing voiceTarget mel-cepstral coefficients extracted from the target
singer’s natural singing voice0

Targetw/o GV (filtered)
w/o GV (estimated)
w/ GV (filtered)

10

-1

10

-2

0

w/ GV (estimated) mel-cepstral coefficient differentials estimated with the proposed DIFFSVC method

w/ GV (estimated)

5

10

15

20

25

Order of mel-cepstrum
Figure 3: GVs of several mel-cepstral sequences.

w/ GV (filtered) mel-cepstral coefficients extracted from thesinging voice converted in the proposed DIFFSVC
methodmethod “w/o GV (filtered)” significantly decreases comparedto that of “Target.” On the other hand, the GV in the proposed
method “w/ GV (filtered)” is close to that of “Target.” This GV
restoration yields significant improvements in speech quality of
the converted singing voice. Note that the GV of the feature
differential trajectories in the proposed method (w/ GV (estimated)) are still similar to those of the conventional method(w/o GV (estimated)). This shows the effectiveness of the proposed method modeling not the GV of the differential trajectory
but the GV of the converted trajectory.

w/o GV (estimated) mel-cepstral coefficient differentials estimated with the conventional DIFFSVC
w/o GV (filtered) mel-cepstral coefficients extracted from thesinging voice converted in the conventional DIFFSVC
methodFigure 2 shows the individual trajectories and the logarithmic F0 trajectory. It can be observed from “Source” and“Target” that higher-order mel-cepstral coefficients tend to have
rapidly varying fluctuations. It has been reported in [18] thatthese fluctuations are well modeled by the modulation spectrum
and strongly affect speech quality of the converted speech. In
the proposed method (w/ GV (estimated)), the converted feature differential trajectory is smoothly connected from the endof voiced segments to the start of voiced frames thanks to the
proposed smoothing process at unvoiced frames. This yields
a converted feature trajectory (w/ GV (filtered)) maintaining
natural fluctuations at unvoiced frames. On the other hand,these fluctuations are obviously reduced in the conventional
method (w/o GV (filtered)). We can also see that the GV of
the converted feature trajectory at higher-order mel-cepstral coefficients is restored more effectively by the proposed method(w/ GV (filtered)) compared to the conventional method (w/o
GV (filtered)). These results imply that the proposed method
effectively approximates the target spectral fluctuations by using those of the source spectral trajectory and the GV of thetarget spectral trajectory.
Figure 3 shows the GVs calculated from several trajectories of mel-cepstral coefficients. The GV in the conventional

5. Conclusions
In order to improve quality of singing voice conversion based on
direct waveform modification (DIFFSVC), we have proposed
DIFFSVC considering global variance and smoothing of theconversion function at unvoiced frames. The experimental results have demonstrated that the proposed DIFFSVC method
makes it possible to convert voice timbre of a source singer intothat of a target singer with higher speech quality while not causing any adverse effects on the conversion accuracy of speaker
identity compared to the conventional DIFFSVC method. In future work, we plan to apply the DIFFSVC framework to crossgender conversion.

6. Acknowledgements
Part of this work was supported by JSPS KAKENHI Grant
Numbers: 26280060 and 15H02726, and by the JST OngaCREST project.

2757

7. References[1] H. Kenmochi and H. Ohshita, “VOCALOID – Commericial
singing synthesizer based on sample concatenation,” Proc. INTERSPEECH, pp. 4011–4012, Aug. 2007.
[2] K. Saino, M. Tachibana, and H. Kenmochi, “A singing stylemodeling system for singing voice synthesizers.” Proc. INTERSPEECH, pp. 2894–2897, Sept. 2010.
[3] K. Oura, A. Mase, T. Yamada, S. Muto, Y. Nankaku, andK. Tokuda, “Recent development of the HMM-based singing
voice synthesis system - Sinsy,” SSW7, pp. 211–216, Sept. 2010.
[4] T. Nakano and M. Goto, “VocaListener: A singing-to-singing synthesis system based on iterative parameter estimation,” Proc. SMC
2009, pp. 343–348, July 2009.
[5] T. Nakano and M. Goto, “Vocalistener2: A singing synthesis
system able to mimic a user’s singing in terms of voice timbrechanges as well as pitch and dynamics,” Proc. ICASSP, pp. 453–
456, May 2011.
[6] F. Villavicencio and J. Bonada, “Applying voice conversion to
concatenative singing-voice synthesis,” Proc. INTERSPEECH,
pp. 2162–2165, Sept. 2010.
[7] Y. Kawakami, H. Banno, and F. Itakura, “GMM voice conversionof singing voice using vocal tract area function,” IEICE technicalreport. Speech (Japanese edition), vol. 110, no. 297, pp. 71–76,
Nov. 2010.
[8] Y. Stylianou, O. Cappé, and E. Moulines, “Continuous probabilistic transform for voice conversion,” IEEE Trans. SAP, vol. 6,
no. 2, pp. 131–142, Mar. 1998.
[9] T. Toda, A. W. Black, and K. Tokuda, “Voice conversion based on
maximum likelihood estimation of spectral parameter trajectory,”
IEEE Trans. ASLP, vol. 15, no. 8, pp. 2222–2235, Nov. 2007.
[10] T. Toda, Y. Ohtani, and K. Shikano, “One-to-many and many-toone voice conversion based on eigenvoices,” Proc. ICASSP, pp.
1249–1252, Apr. 2007.
[11] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Many-tomany eigenvoice conversion with reference voice,” Proc. INTERSPEECH, pp. 1623–1626, Sept. 2009.
[12] H. Doi, T. Toda, T. Nakano, M. Goto, and S. Nakamura, “Singing
voice conversion method based on many-to-many eigenvoice conversion and training data generation using a singing-to-singing
synthesis system,” Proc. APSIPA ASC, Nov. 2012.
[13] K. Kobayashi, T. Toda, G. Neubig, S. Sakti, and S. Nakamura,“Statistical singing voice conversion with direct waveform modification based on the spectrum differential,” Proc. INTERSPEECH,
pp. 2514–2418, Sept. 2014.
[14] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura, “Speech parameter generation algorithms for HMM-based
speech synthesis,” Proc. ICASSP, pp. 1315–1318, June 2000.
[15] M. Sambur, “Selection of acoustic features for speaker identification,” IEEE Trans. ASSP, vol. 23, no. 2, pp. 176–182, 1975.
[16] H. Kawahara, I. Masuda-Katsuse, and A. Cheveigné, “Restructuring speech representations using a pitch-adaptive time-frequency
smoothing and an instantaneous-frequency-based f0 extraction:
Possible role of a repetitive structure in sounds,” Speech Communication, vol. 27, no. 3-4, pp. 187–207, Apr. 1999.
[17] S. Imai, K. Sumita, and C. Furuichi, “Mel log spectrum approximation (mlsa) filter for speech synthesis,” Electronics and Communications in Japan (Part I: Communications), vol. 66, no. 2,
pp. 10–18, 1983.
[18] S. Takamichi, T. Toda, A. Black, and S. Nakamura, “Modulation
spectrum-based post-filter for gmm-based voice conversion,” APSIPA ASC, Dec. 2014.

2758