Gender-dependent Spectrum Diﬀerential Models for
Perceived Age Control based on Direct Waveform
Modification in Singing Voice Conversion
Kazuhiro Kobayashi∗ , Tomoki Toda∗ , Tomoyasu Nakano† , Masataka Goto† ,Graham Neubig∗ , Sakriani Sakti∗ and Satoshi Nakamura∗∗ Graduate School of Information Science, Nara Institute of Science and Technology (NAIST)

Takayama 8916–5, Ikoma, Nara, 630–0192 Japan† National Institute of Advanced Industrial Science and Technology (AIST)

1-1-1 Umezono, Tsukuba, Ibaraki 305-8568 Japan

Abstract—The perceived age of a singing voice, which is the ageof the singer as perceived by the listener, is one of the intuitively
understandable measures to describe voice characteristics of thesinging voice. Singers can sing expressively by controlling voicetimbre to some extent but the varieties of voice timbre that singers
can produce are limited by physical constraints. To overcome thislimitation, previous work has proposed statistical voice timbre
control technique based on the perceived age. This techniquemakes it possible to control the perceived age of singing voice
while retaining singer individuality by the use of statistical voice
conversion (SVC) with a multiple-regression Gaussian mixture
model (MR-GMM). However, the range of controllable perceived
age is limited and speech quality of the converted singing voice
is significantly degraded compared to that of a natural singing
voice. In this paper, we propose a method for perceived age
control using direct waveform modification based on spectrum
diﬀerential and gender-dependent modeling. The experimental
results show that the proposed method makes the range ofcontrollable perceived age wider and quality of converted singing
voice higher compared to the conventional method.

I. Introduction
The singing voice is one of the most expressive components
in music. In addition to pitch, dynamics, and rhythm, singers
can express various expressions by using the linguistic information of the lyrics. However, singers usually have diﬃculty inchanging their voice timbre widely due to physical constraints
in speech production. If it would be possible to freely control
voice timbre of singers beyond their physical constraints, itwill open up entirely new forms of expression in music.
Several techniques to control the voice timbre of the singing
voice have been proposed. One approach is based on speechmorphing [1] in the speech analysis/synthesis framework [2].
Another approach is based on statistical voice conversion
techniques [3], [4]. A Gaussian mixture model (GMM) andan eigenvoice GMM (EV-GMM) [5] have been successfully
applied to singing voice conversion (SVC) that converts thesource singer’s timbre into another target singer’s timbre [6],[7], [8]. In particular, SVC with the EV-GMM makes it
possible to convert singing voice timbre of an arbitrary source
singer into that of an arbitrary target singer in any song byautomatically adapting voice timbre control parameters to thegiven singing voices of those singers. However, it is still
diﬃcult to achieve the desired timbre if no target singer’s
singing voice is available because it is hard to predict thechange of voice timbre caused by manually manipulating each
adaptive parameter.
In our previous work [9], we have proposed a method
for controlling one aspect of the singing voice, perceived

978-616-361-823-8 © 2014 APSIPA

age, based on a multiple-regression GMM (MR-GMM) [10].
However, the range of controllable perceived age is still
limited. This is possibly caused by using a single MR-GMM
to model all singers’ voice timbre because it has been reported
that spectral variations caused by aging are diﬀerent between
male and female speakers [11], [12]. Additionally, quality of
the converted singing voice tends to be significantly degraded
compared to that of a natural singing voice. The use of a
vocoder to synthesize the converted singing voice is one of
the biggest factors causing this degradation. To address thedegradation problem by the use of the vocoder, a statisticalconversion method based on direct waveform modification has
been proposed [13]. The statistical conversion method avoids
the degradation by directly filtering a waveform of the input
singing voice based on time-varying spectrum diﬀerential that
estimated with the traditional GMM.
In this paper, we improve the controllability of perceived
age and the quality of the converted singing voice in ourpreviously proposed voice timbre control technique based on
the perceived age. To make the range of controllable perceived
age wider, we propose a voice timbre control technique withgender-dependent MR-GMMs that can more accurately model
the spectral variations caused by a change of the perceived
age in each gender. Furthermore, we also apply the statisticalconversion method based on direct waveform modification to
the voice timbre control technique based on perceived age. It isshown from results of subjective evaluation that the proposed
methods significantly improve performance of the perceived
age control compared to the conventional method.
II. Voice timbre control based on perceived age while
retaining singer individuality
A. Training process
1) Training of the MR-GMM: The MR-GMM is trainedusing multiple parallel data sets consisting of the reference singer’s singing voices and many pre-stored target
singers’ singing voices. The joint probability density of 2Ddimensional joint static and dynamic feature vectors modeledby the MR-GMM is given by"!
(MR), w(s)P Xt , Y (s)t |λ& % (XX)
& %
&'
$%
M
(X)
#
Xt
Σm
µm
Σ(XY)
m
,
;
, (1)
αm N
=
X)
Y (s)
µ(Y)
Σ(Y
Σ(YY)
t
m (s)
m
m
m=1
(Y)
(Y) (s)µ(Y)m (s) = bm w + µm ,(2)

APSIPA 2014

where N (·; µ, Σ) denotes the normal distribution with a mean
vector µ and a covariance matrix Σ. The mixture component
index is m. The total number of mixture components is M. The(s)⊤vectors Xt = [x⊤t , ∆x⊤t ]⊤ and Y (s), ∆Y (s)⊤]⊤ are jointt = [Y ttstatic and delta feature vectors of the reference singer and thes-th pre-stored target singer, which are automatically alignedby dynamic time warping for their corresponding singing(Y)voices. The vectors b(Y)m and µm indicate the representative
vector and bias vector, respectively. The value w(s) indicatesthe perceived age score of the s-th pre-stored target singer,
which is manually assigned to each pre-stored target singer.
The notation λ(MR) indicates an MR-GMM parameter set consisting of mixture-dependent parameters, such as the mixturecomponent weight αm , the mean vector µ(X)m , the representative(Y)
vector b(Y)m , the bias vector µm and the covariance matrix Σmof the m-th mixture component.
λ(MR) is a MR-GMM parameter set consisting of the
mixture-component weight αm , the mean vector µm , the repre(Y)sentative vector b(Y)m , the bias vector µm , and the covariancematrix Σm of the m-th mixture component.
To easily create the MR-GMMs for various source singers(i.e., users), the framework of the many-to-many SVC [8] isapplied to the MR-GMM for the reference singer.
The joint probability density of many-to-many MR-GMM
is given by"!
(o) (MR), w(i) , w(o)P Y (i)t , Y t |λM#
" ( ! (i)"!
P Y t |Xt , m, λ(MR) , w(i)P m|λ(MR)
=
m=1

" !
"
!
(MR)
, w(o) P Xt |m, λ(MR) dXt
P Y (o)t |X t , m, λ
& % (YY)&'$% (i) & % (Y)
M
XY)
#
Σm
µm (i)
Σ(Y
Yt
m
,
;
, (3)
=
αm N
(o)
(Y)
(Y XY)
(YY)
Yt
Σm
Σm
µm (o)m=1−1

XY)X) (XX)
Σ(Y
= Σ(Y
Σ(XY)
m
m Σm
m ,
(i)

(4)

(o)

where w and w indicate the perceived age scores of thesource singer and target singers, respectively. The source and(Y)target mean vectors, µ(Y)m (i) and µm (o) are given by Eq. (2).
An MR-GMM capable of converting each source singer canbe easily created by adapting the perceived age score of thesource singer.
2) Adaptation of MR-GMMs to a specific source singer:
To implement the perceived age control while retaining source
singer’s individuality, we modify the representative form of
the target mean vectors. The perceived age score of the target
singing voice w(o) can be represented by that of the source
singer w(i) and a perceived age score diﬀerential ∆w, i.e., w(o) =w(i) + ∆w. Then, the target mean vector of the m-th mixture
component is given by(Y)(Y) (i)(Y)µ(Y)
m (o) = bm w + µm + bm ∆w,

(5)

(Y)
(i)
where b(Y)m w + µm can be regarded as an approximated formof the source mean vector. Therefore, it is modified as follows:
µ(Y)
m (o)
µ̂(Y)
m

≃

(Y)
µ̂(Y)
m + bm ∆w,(6)is the maximum likelihood estimate estimated
whereby the use of parallel data of source and reference singers.
Consequently, the source and target mean vectors of themodified MR-GMM for the source singer are represented by(Y)
the source mean vectors µ̂(Y)m , the representative vectors bm ,and the perceived age score diﬀerential ∆w.

B. Conversion process
In the conversion process, the perceived age score differential ∆w is manually set to a desired value. Then, thesource singer’s singing voice is converted into a perceived age
controlled singing voice using maximum likelihood estimationof the speech parameter trajectory with the modified MRGMM [4].
Time sequence vectors of the source and converted features are denoted as Y(i) = [Y ⊤1 (i), · · · , Y ⊤T (i)]⊤ and Y(o) =[Y ⊤1 (o), · · · , Y ⊤T (o)]⊤ , where T is the number of frames included in the time sequence of the given source feature
vectors. A time sequence vector of the converted static featuresŷ(o) = [ŷ⊤1 (o), · · · , ŷ⊤T (o)]⊤ is determined as follows:ŷ(o) = argmax P(Y(o)|Y(i), λ(MR) , µ̂(Y)m , ∆w)y(o)subject to Y(o) = Wy(o),(7)

where W is a transformation matrix to expand the static feature
vector sequence into the joint static and dynamic feature vector
sequence [14]. The conditional probability density function
P(Y(o)|Y(i), λ(MR) , µ̂(Y)m , ∆w) is analytically derived from themodified MR-GMM. To alleviate the oversmoothing eﬀects
that usually make the converted singing voice sound muﬄed,
global variance (GV) [4] is also considered.
III. Perceived age control based on gender-dependent
MR-GMM with direct waveform modification
To improve controllability and quality of the converted
speech of the conventional perceived age control method,
we further implement two techniques, 1) gender-dependent
MR-GMMs for more accurately capturing spectral variations
depending on the perceived age and 2) direct waveform
modification based on spectral diﬀerential.
A. Gender-dependent MR-GMM
In the conventional method, the MR-GMM is trained using
multiple parallel data sets consisting of singing voice pairsof the reference singer and all pre-stored target singers including both male and female singers. On the other hand, inthe gender-dependent modeling, two MR-GMMs are trained
separately using the parallel data sets consisting of only malesingers or female singers as the reference singer and thepre-stored target singers. To create the MR-GMM for thesource singer, the corresponding gender-dependent MR-GMM
is adapted to him/her to develop the modified MR-GMM in
the same manner as described in Section II.
B. Perceived age control with direct waveform modification
based on spectral diﬀerential
In the direct waveform modification based on spectral
diﬀerential, the spectral feature diﬀerential between the source
singing voice and the converted singing voice is directly
estimated from the source singer’s spectral features using a
diﬀerential MR-GMM (DIFFMR-GMM), which is analyticallyderived from the modified MR-GMM. The joint probability
density of the source singer’s spectral features and the spectralfeature diﬀerential related to perceived age control is modeledusing the DIFFMR-GMM. Then, voice timbre of the source
singer’s singing voice is converted by directly filtering awaveform of the source singer’s natural singing voice withthe time-varying spectral feature diﬀerential determined with
the DIFFMR-GMM. In this conversion process, the converted
singing voice is free from various errors usually observed in
the conventional waveform generation process with vocoder,such as F0 extraction errors, unvoiced/voiced decision errors,
spectral parameterization errors caused by liftering on the melcepstrum, and so on.
is derived as follows. Let Dt =) The DIFFMR-GMM*⊤
d⊤t , ∆d⊤tdenote the joint static and delta diﬀerential feature
vector, where dt = yt (o) − yt (i). The 2D-dimensional joint
static and delta feature vector between the source and thediﬀerential features is represented as linear transformation of
the conventional joint feature vectors as follows:& +% (i) & %, % (i) &
Y (i)
I 0
Yt
Yt
t
=
= −I I
,
(8)
(o)
(i)
Dt
Yt − YtY (o)t

In the conversion process, the converted diﬀerential feature
vector is determined in the same manner as described in Sect.
II-B. In this paper, the GV is not considered in the conversion
process based on the spectrum diﬀerential.

SVC (D): with gender-dependent MR-GMM
DIFFSVC (D): with gender-dependent DIFFMR-GMM
The converted singing voice samples were generated by settings of the perceived age score diﬀerential to -60, -30, 0, 30,
and 60. The number of source singers was 16, who were notincluded in the pre-stored target singers. We used P039 as anevaluation song.
First, we evaluated perceived age controllability. Eight subjects were divided into two groups, and the 16 evaluation
singers were divided into two groups so that one group always
included one male singer and one female singer in each age
group. Each subject evaluated the converted singing voicesfrom only one group of the evaluation singers. Subjects wereasked to assign the perceived age to each converted singing
voice by listening to it in random order.
In the second experiment, we evaluated the quality of theconverted singing voice using a mean opinion score (MOS).
The number of subjects and evaluation singers were the sameas in the first experiment. The subjects rated the quality of
the converted singing voice using a 5–point scale: “5” forexcellent, “4” for good, “3” for fair, “2” for poor, and “1”for bad.
In the final experiment, we conducted an XAB test on thesinger individuality to compare SVC (I) and DIFFSVC (D).
The subjects and evaluation singers were separated into twogroups in the same manner as the first experiment. A pair of
singing voices converted by SVC (I) and by DIFFSVC (D) for
the same singer and a setting of the perceived age score waspresented to the subjects after presenting the natural singing
voice as a reference. Then, they were asked which singing
voice sounded more similar to the reference in terms of thesinger individuality.

IV. Experimental evaluations
A. Experimental conditions
We used the AIST humming database [15] consisting ofsongs with Japanese lyrics sung by Japanese male and female
amateur singers in their 20s, 30s, 40s, and 50s. The sampling
frequency was set to 16 kHz. The 1st through 24th mel-cepstralcoeﬃcients extracted by STRAIGHT analysis [16] were usedas spectral features. As the source excitation features, we used
F0 and aperiodic components in five frequency bands, i.e., 0–
1, 1–2, 2–4, 4–6, and 6–8 kHz, which were also extracted
by STRAIGHT analysis [17]. The frame shift was 5 ms. The
mel log spectrum approximation filter [18] was used as thesynthesis filter in both the conventional waveform generation
with vocoder and the proposed direct waveform modification.
In the training of the gender-independent MR-GMM, weused parallel data sets of a female reference singer in her
20s and 56 pre-stored target singers including 28 males and
28 females in their 20s, 30s, 40s and 50s. In the trainingof the gender-dependent MR-GMMs, we separately used afemale and male reference singer in their 20s and 28 male or
28 female pre-stored target singers. The number of mixture
components of each MR-GMM was 256 for the spectralfeature and 128 for the aperiodic components. The numberof training songs was 23 for each singer. The duration of eachsong was approximately 20 seconds. The perceived age scorefor each singer was determined as an average score of thesinger rated by 8 subjects in their 20s [9].
In this evaluation, we define several methods follow:• SVC (I): with gender-independent MR-GMM

B. Experimental Results
Figure 1 shows the relationship between settings of theperceived age diﬀerential and actually perceived age of theconverted singing voice in each method. We can see that theperceived age varies almost linearly according to a changeof the settings of the perceived age diﬀerential from -60 to
60. Moreover, a range of the perceived age of the converted
singing voice becomes wider by using SVC (D) and DIFFSVC(D) compared to SVC (I). This indicates that the genderdependent model is eﬀective for accurately modeling spectral
variations depending on the perceived age.
Figure 2 indicates the results of the opinion test on thequality. We can see that DIFFSVC (D) tends to significantly
improve quality of the converted singing voices compared to
SVC (I) and SVC (D). Although the quality is greatly degradedin the conventional method SVC (I) as the perceived age
score diﬀerential is set to larger or smaller values, this quality
degradation is eﬀectively alleviated by the proposed method
DIFFSVC (D).
Figure 3 indicates the result of the XAB test on the singer
individuality. DIFFSVC (D) better or equally retains singer
individuality in these perceived age settings compared to theconventional method SVC (I). We can see that as a change ofthe perceived age diﬀerential setting is larger, the diﬀerence
between DIFFSVC (D) and SVC (I) becomes smaller.
These results suggest that 1) the gender-dependent MRGMM is eﬀective for improving the perceived age controllability, and 2) the direct waveform modification techniquewith spectral diﬀerential significantly improves quality of theconverted singing voice.

where I denotes the identity matrix. Applying this linear
transform to the modified MR-GMM, the DIFFMR-GMM isderived as follows:"!
(MR) (Y), µ̂m , ∆w
P Y (i)t , Dt |λ& % (YY) (DY D) &'$% (i) & %
M
#
Σm Σm
µ̂(Y)
Yt
m
,
;
, (9)
αm N
=
(DY D) (DD)
(Y)
Dt
b
∆w
Σ
Σm
m
m
m=1
(DY D)
XY)
(DD)
XY)
= Σ(Y
− Σ(YY)= 2(Σ(YY)
− Σ(Y
).
Σm
m
m , Σm
m
m

(10)

•
•

Difference in perceived age
after conversion

10
5
0
-5
-10
10
5
0
-5
-10
10
5
0
-5
-10

age wider and 2) it also enables to significantly improve
quality of the converted singing voice. In future work, we willconsider adaptive techniques for an arbitrary source singer by
the use of several singing voices.

SVC (I)
95% confidence interval
Regression line

Acknowledgment
This work was supported in part by JSPS KAKENHI
Grant Numbers: 26280060 and 25280061 and by the JST
OngaCREST project.

SVC (D)DIFFSVC (D)

-60

-30

References
0

30

60

Perceived age conversion setting
Fig. 1. Setting and actual diﬀerential in perceived age after conversion.

Mean opinion score

5
4.5
Natural singing voice

4

SVC (I)
SVC (D)DIFFSVC (D)
95% confidence interval

3.5
3
2.5
2
1.5
1

-60

-30

0

30

60

Perceived age conversion setting
Fig. 2. Mean opinion score of speech quality.

Preference score [%]

100
80
60
40
SVC (I)DIFFSVC (D)
95% confidence interval

20
0

-60

-30

0

30

60

Perceived age conversion setting
Fig. 3. Comparing singer individuality.

V. Conclusions
To improve performance of our previously proposed perceived age control technique based on multiple-regression
Gaussian mixture models (MR-GMM), we have successfully
implemented a gender-dependent modeling technique and a
direct waveform modification technique with spectral diﬀerential. The experimental results have demonstrated that 1) the
proposed method makes a range of the controllable perceived[1] H. Kawahara, R. Nisimura, T. Irino, M. Morise, T. Takahashi, and
H. Banno, “Temporally variable multi-aspect auditory morphing enabling extrapolation without objective and perceptual breakdown,” Proc.
ICASSP, pp. 3905–3908, Apr. 2009.
[2] M. Morise, M. Onishi, H. Kawahara, and H. Katayose, “v. morish’ 09:
A morphing-based singing design interface for vocal melodies,” Proc.
ICEC, pp. 185–190, 2009.
[3] Y. Stylianou, O. Cappé, and E. Moulines, “Continuous probabilistictransform for voice conversion,” IEEE Trans. SAP, vol. 6, no. 2, pp.
131–142, Mar. 1998.
[4] T. Toda, A. W. Black, and K. Tokuda, “Voice conversion based on
maximum likelihood estimation of spectral parameter trajectory,” IEEE
Trans. ASLP, vol. 15, no. 8, pp. 2222–2235, Nov. 2007.
[5] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Many-to-many
eigenvoice conversion with reference voice,” Proc. INTERSPEECH, pp.
1623–1626, Sept. 2009.
[6] F. Villavicencio and J. Bonada, “Applying voice conversion to concatenative singing-voice synthesis,” Proc. INTERSPEECH, pp. 2162–2165,
Sept. 2010.
[7] Y. Kawakami, H. Banno, and F. Itakura, “GMM voice conversion of
singing voice using vocal tract area function,” IEICE technical report.
Speech (Japanese edition), vol. 110, no. 297, pp. 71–76, Nov. 2010.
[8] H. Doi, T. Toda, T. Nakano, M. Goto, and S. Nakamura, “Singing voice
conversion method based on many-to-many eigenvoice conversion andtraining data generation using a singing-to-singing synthesis system,”
Proc. APSIPA ASC, Nov. 2012.
[9] K. Kobayashi, T. Toda, H. Doi, T. Nakano, M. Goto, G. Neubig, S. Sakti,and S. Nakamura, “Voice timbre control based on perceived age insinging voice conversion,” IEICE Trans. Inf. Syst., vol. E97-D, no. 6,
pp. 1419–1428, June 2014.
[10] K. Ohta, T. Toda, Y. Ohtani, H. Saruwatari, and K. Shikano, “Adaptive
voice-quality control based on one-to-many eigenvoice conversion,”
Proc. INTERSPEECH, pp. 2158–2161, Sept. 2010.
[11] W. Endres, W. Bambach, and G. Flsser, “Voice spectrograms as afunction of age, voice disguise, and voice imitation,” The Journal of the
Acoustical Society of America, vol. 49, no. 6B, pp. 1842–1848, 1971.
[12] S. E. Linville and J. Rens, “Vocal tract resonance analysis of aging
voice using long-term average spectra,” Journal of Voice, vol. 15, no.
3, pp. 323–330, 2001.
[13] K. Kobayashi, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, “Statistical singing voice conversion with direct waveform modification basedon the spectrum diﬀerential,” Proc. INTERSPEECH, Sept. 2014.
[14] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura,“Speech parameter generation algorithms for HMM-based speech synthesis,” Proc. ICASSP, pp. 1315–1318, June 2000.
[15] M. Goto and T. Nishimura, “AIST humming database: Music databasefor singing research,” IPSJ SIG Notes (Technical Report) (Japaneseedition), vol. 2005-MUS-61-2, no. 82, pp. 7–12, Aug. 2005.
[16] H. Kawahara, I. Masuda-Katsuse, and A. Cheveigné, “Restructuring
speech representations using a pitch-adaptive time-frequency smoothing
and an instantaneous-frequency-based f0 extraction: Possible role of arepetitive structure in sounds,” Speech Communication, vol. 27, no. 3-4,
pp. 187–207, Apr. 1999.
[17] H. Kawahara, J. Estill, and O. Fujimura, “Aperiodicity extraction andcontrol using mixed mode excitation and group delay manipulation fora high quality speech analysis, modification and system straight,” Proc.
MAVEBA, Sept. 2001.
[18] S. Imai, K. Sumita, and C. Furuichi, “Mel log spectrum approximation(MLSA) filter for speech synthesis,” Electronics and Communicationsin Japan (Part I: Communications), vol. 66, no. 2, pp. 10–18, 1983.