INTERSPEECH 2013

An Investigation of Acoustic Features for Singing Voice Conversion based on
Perceptual Age
Kazuhiro Kobayashi1 , Hironori Doi1 , Tomoki Toda1 , Tomoyasu Nakano2 , Masataka Goto2 ,Graham Neubig1 , Sakriani Sakti1 , Satoshi Nakamura1
Graduate School of Information Science, Nara Institute of Science and Technology (NAIST), Japan1
National Institute of Advanced Industrial Science and Technology (AIST), Japan2{kazuhiro-k, hironori-d, tomoki, neubig, ssakti, s-nakamura}@is.naist.jp1{t.nakano, m.goto}@aist.go.jp2

Abstract

morphing can only be applied to singing voice samples of thesame song.

In this paper, we investigate the acoustic features that canbe modified to control the perceptual age of a singing voice.
Singers can sing expressively by controlling prosody and vocal
timbre, but the varieties of voices that singers can produce arelimited by physical constraints. Previous work has attempted
to overcome this limitation through the use of statistical voice
conversion. This technique makes it possible to convert singing
voice characteristics of an arbitrary source singer into those ofan arbitrary target singer. However, it is still difficult to intuitively control singing voice characteristics by manipulating parameters corresponding to specific physical traits, such as gender and age. In this paper, we focus on controlling the perceived
age of the singer and, as a first step, perform an investigation of
the factors that play a part in the listener’s perception of thesinger’s age. The experimental results demonstrate that 1) theperceptual age of singing voices corresponds relatively well tothe actual age of the singer, 2) speech analysis/synthesis processing and statistical voice conversion processing don’t cause
adverse effects on the perceptual age of singing voices, and 3)
prosodic features have a larger effect on the perceptual age thanspectral features.
Index Terms: singing voice, voice conversion, perceptual age,
spectral and prosodic features, subjective evaluations.

To make it possible to more flexibly change of singing voice
characteristics, statistical VC techniques [2, 3] have been successfully applied to convert the source singer’s singing voice
into another target singer’s singing voice [4, 5]. In this method,a conversion model is trained in advance using acoustic features, which are extracted from a parallel data set of song pairssung by the source and target singers. The trained conversion
model makes it possible to convert the acoustic features of thesource singer’s singing voice into those of the target singer’s
singing voice in any song, keeping the linguistic information of
the lyrics unchanged. Furthermore, to develop a more flexible
singing VC system, eigenvoice conversion (EVC) techniques[6] have been applied to singing VC [7]. In a singing VC system based on many-to-many EVC [8], which is one particularvariety of EVC, an initial conversion model called the canonical eigenvoice GMM (EV-GMM) is trained in advance using
multiple parallel data sets including song pairs of a single reference singer and many other singers. The EV-GMM is adapted
into arbitrary source and target singers by automatically estimating a few adaptive parameters from the given singing voicesamples of those singers. Although this system is also capableof flexibly changing singing voice characteristics by manipulating the adaptive parameters even if no target singing voicesample is available, it is difficult to achieve the desired singing
voice characteristics, because it is hard to predict the changeof singing characteristics caused by the manipulation of each
adaptive parameter.

1. Introduction
The singing voice is one of the most expressive components inmusic. In addition to pitch, dynamics, and rhythm, the linguistic
information of the lyrics can be used by singers to express morevarieties of expression than other music instruments. Although
singers can also expressively control their voice characteristics
such as voice timbre to some degree, they usually have difficultyin changing their own voice characteristics widely, (e.g. changing them into those of another singer’s singing voice) owing tophysical constraints in speech production. If it would be possible for singers to freely control voice characteristics beyond
these physical constraints, it will open up entirely new ways for
singers to express themselves.
In previous research, a number of techniques have been
proposed to change the characteristics of singing voices. One
typical method is singing voice conversion (VC) based on
speech morphing in the speech analysis/synthesis framework[1]. This method makes it possible to independently morph several acoustic parameters, such as spectral envelope, F0 , and duration, between singing voices of different singers or different
singing styles. One of the limitations of this method is that the

Copyright © 2013 ISCA

In the area of statistical parametric speech synthesis [9],there have been several attempts at developing techniques formanually controlling voice quality of synthetic speech by manipulating intuitively controllable parameters corresponding to
specific physical traits, such as gender and age. Nose et al.
[10] proposed a method for controlling speaking styles in synthetic speech with multiple regression hidden Markov models(HMM). Tachibana et al. [11] extended this method to control voice quality of synthetic speech using a voice quality control vector assigned to expressive word pairs describing voice
quality, such as “warm – cold” and “smooth – non-smooth”. A
similar method has also been proposed in statistical VC [12].
Although these methods have only been applied to voice quality control of normal speech, it is expected that they would also
be effective for controlling singing voice characteristics.
In this paper, we focus on the perceptual age, or the agethat a listener predicts the singer to be, of singing voices as one1057

25- 29 August 2013, Lyon, France

of the factors to intuitively describe the singing voice. For normal speech, there is some research investigating acoustic featurechanges caused by aging. It has been reported that aperiodicity
of excitation signals tends to increase with aging [13]. A perceptual age classification method to classify speech of elderly
people and non-elderly people using spectral and prosodic features has also been developed [14]. On the other hand, the perceptual age of singing voices has not yet been studied deeply.
As fully understanding the acoustic features that contributeto the perceptual age of singing voices is essential to the development of VC techniques to modify a singer’s perceptual
age, in this paper we perform an investigation of the acoustic features that play a part in the listener’s perception of thesinger’s age. We conduct several types of perceptual evaluation
to investigate 1) how well the perceptual age of singing voices
corresponds to the actual age of the singer, 2) whether or not
singing VC processing causes adverse effects on the perceptual
age of singing voices, and 3) whether spectral or prosodic features have a larger effect on the perceptual age.

joint probability density given by Eq. (1). To alleviate the oversmoothing effects that usually make the converted speech soundmuffled, global variance (GV) [3] is also considered in conversion.

3. Investigation of acoustic features
affecting perceptual age
In the traditional SVC [5, 7], only the spectral features such as
mel-cepstrum are converted. It is straightforward to also convert
the aperiodic components [16], which capture noise strength oneach frequency band of the excitation signal, as in the traditional
VC for natural voices [17]. If the perceptual age of singing
voices is captured well by these acoustic features, it will make itpossible to develop a real-time SVC system capable of controlling the perceptual age of singing voices by combining the voice
quality control based on statistical VC [12] and real-time statistical VC techniques [18, 19]. On the other hand, if the perceptual age of singing voices is not captured well by these acoustic
features, which mainly represent segmental features, the conversion of other acoustic features, such as prosodic features(e.g., F0 pattern), will also be necessary. In such a case, thevoice-quality control framework of HMM-based speech synthesis [10, 11] can be used in the SVC system to control the perceptual age of singing voices, although it is not straightforward
to develop a real-time SVC system in this framework. Because
the synthesis technique that must be used will change according
to the acoustic features to be converted, it will be highly beneficial to make clear which acoustic features need to be modified to control the perceptual age of singing voices. To do so,
we compare the perceptual age of natural singing voices withthat of several types of synthesized singing voices by modifying acoustic features as shown in Table 1.

2. Statistical singing voice conversion
Statistical singing VC (SVC) consists of a training process anda conversion process. In the training process, a joint probability density function of acoustic features of the source andtarget singers’ singing voices is modeled with a GMM using
a parallel data set in the same manner as in statistical VC fornormal voices [5]. As the acoustic features of the source andtarget singers, we employ 2D-dimensional joint static and dy⊤ ⊤namic feature vectors X t = [x⊤of the source andt , ∆xt ]⊤⊤ ⊤Y t = [y t , ∆y t ] of the target consisting of D-dimensional
static feature vectors xt and y t and their dynamic feature vectors ∆xt and ∆y t at frame t, respectively, where ⊤ denotes
the transposition of the vector. Their joint probability density
modeled by the GMM is given by
P (X t , Y t |λ)([ ][][ (XX) (XY ) ])M∑
X t µ(X)
Σm Σm
m
=
αm N
;
, (1)
) ,
(Y X)
(Y Y )
Y t µ(Y
Σm Σm
m
m=1

3.1. Analysis/synthesis with aperiodic components (w/ AC)In the analysis/synthesis framework, a voice is first converted
into parameters of the synthesis model described in Section 2,
then simply re-synthesized into a waveform using these parameters without change. As analysis and synthesis are necessary
steps in converting acoustic features of singing voices, we investigate the effects of distortion caused by analysis/synthesison the perceptual age of singing voices. STRAIGHT [20] is a
widely used high-quality analysis/synthesis method, so we use
it to extract acoustic features consisting of mel-cepstrum, F0 ,and aperiodic components.

where N (·; µ, Σ) denotes the normal distribution with a mean
vector µ and a covariance matrix Σ. The mixture component
index is m. The total number of mixture components is M . λis a GMM parameter set consisting of the mixture-component
weight αm , the mean vector µm , and the covariance matrix
Σm of the m-th mixture component. A GMM is trained using
joint vectors of X t and Y t in the parallel data set, which areautomatically aligned to each other by dynamic time warping.
In the conversion process, the source singer’s singing voice
is converted into the target singer’s singing voice with the GMM
using maximum likelihood estimation of speech parameter trajectory [3]. Time sequence vectors of the source features and⊤ ⊤the target features are denoted as X = [X ⊤and
1 , · · · , XT ]⊤⊤ ⊤Y = [Y 1 , · · · , Y T ] where T is the number of frames included in the time sequence of the given source feature vectors. A time sequence vector of the converted static features⊤ ⊤ŷ = [ŷ ⊤1 , · · · , ŷ T ] is determined as follows:ŷ = argmax P (Y |X, λ) subject to Y = W y,y3.2. Analysis/synthesis without aperiodic components (w/oAC)
As mentioned above, previous research [13] has shown that aperiodic components tend to change with aging in normal speech
as mentioned above. We investigate the effects of aperiodic
components on the perceptual age of singing voices. Analysis/synthesized singing voice samples are reconstructed frommel-cepstrum and F0 extracted with STRAIGHT. In synthesis, only a pulse train with phase manipulation [20] instead of
STRAIGHT mixed excitation [17] is used to generate voiced
excitation signals.

(2)
3.3. Intra-singer SVC

where W is a transformation matrix to expand the static feature vector sequence into the joint static and dynamic feature
vector sequence [15]. The conditional probability density function P (Y |X, λ) is analytically derived from the GMM of the

In SVC, conversion errors are inevitable. For example, somedetailed structures of acoustic features not well modeled by
the GMM of the joint probability density and often disappear
through the statistical conversion process. Therefore, the acous-

1058

Features
Mel-cepstrum
Aperiodic components
Power, F0 , duration

Table 1: Acoustic features of several types of synthesized singing voices.
Analysis/synthesis(w/o AC)
Intra-singer SVC
Source singer
Converted to source singer
None
Converted to source singer
Source singer
Source singer

Analysis/synthesis (w/ AC)
Source singer
Source singer
Source singer

each song was approximately 20 seconds. One Japanese malesubject was asked to guess the age of each singing voice by listening to it.
In the second experiment, we investigate the acoustic features that affect the perceptual age of singing voices, by comparing the perceptual age of natural singing voices with thatof each type of synthesized singing voice as shown in Table 1.
Eight Japanese male subjects in their 20s assigned a perceptual
age to each synthesized singing voice. To reduce the subjects’burden, one Japanese song (No. 39) that showed the highest
correlation between the perceptual age and the actual age in thefirst evaluation was selected to be evaluated. Moreover, we selected 16 singers consisting of four singers (two male singers
and two female singers) from each age group, i.e., their 20s, 30s,
40s, or 50s, who showed good correlation between the perceptual age and their actual age. The subjects were separated into
two groups, A and B. The singers were also separated into twogroups, A and B, so that one group always includes one malesinger and one female singer in each age group. The subjects ineach group evaluated only singing voices of the corresponding
singer group.
The sampling frequency was set to 16 kHz. The 1st through24th mel-cepstral coefficients extracted by STRAIGHT analysis
were used as spectral features. As the source excitation features,
we used F0 and aperiodic components in five frequency bands,i.e., 0–1, 1–2, 2–4, 4–6, and 6–8 kHz, which were also extracted
by STRAIGHT analysis. The frame shift was 5 ms.
As training data for the GMMs used in intra-singer SVC
and SVC, we used 18 songs including the evaluation song (No.
39). In the intra-singer SVC, GMMs for converting the melcepstrum and aperiodic components were trained for each of
the selected 16 singers. Another singer not included in these
16 singers was used as the reference singer to create each parallel data set for the GMM training. In the SVC, the GMMs
for converting mel-cepstrum and aperiodic components weretrained for all combinations of the source and target singer pairs
in each singer group. The numbers of mixture components ofeach GMM were optimized experimentally.

tic space on which the converted acoustic features are distributed tends to be smaller than the acoustic space that of thenatural acoustic features. We investigate the effect of the conversion errors caused by this acoustic space reduction on theperceptual age of singing voices by converting one singer’s
singing voice into the same singer’s singing voice. This SVC
process is called intra-singer SVC in this paper.
To achieve intra-singer SVC for a specific singer, we must
create a GMM to model the joint probability density of thesame singer’s acoustic features, i.e., P (X t , X ′t |λ) where X t
and X ′t respectively show the source and target acoustic features of the same singer, needs to be developed. Note thatX t is different from X ′t , they depend on each other, and bothare identically distributed. This GMM is analytically derived
from the GMM of the joint probability density of the acoustic features of the same singer and another reference singer,i.e., P (X t , Y t |λ) where X t and Y t respectively show thesource feature vector of the same singer and that of the reference singer, by marginalizing out the acoustic features of thereference singer in the same manner as used in the many-tomany EVC [7, 8] as follows:
∫
M
∑
(
)
P X t , X ′t |λ =P (m|λ) P (X t |Y t , m, λ)
m=1()
P X ′t |Y t , m, λ P (Y t |m, λ) dY t([ ][][ (XX) (XY X) ])M∑
X t µ(X)
Σm
Σm
m
=
αm N
;
, (XY
, (3)
X)
(XX)
X ′t µ(X)
Σ
Σm
m
m
m=1
X)
) (Y Y )Σ(XY= Σ(XY
Σm
m
m

−1

X)
Σ(Y
.
m

SVC
Converted to target singer
Converted to target singer
Source singer(4)

Using this GMM, intra-singer SVC is performed in the samemanner as described in Section 2. The converted singing voicesample essentially has the same singing voice characteristics as
those before the conversion although they suffer from conversion errors.
3.4. SVC
To investigate which acoustic features have a larger effect on theperceptual age of singing voices, segmental features or prosodic
features, we use the SVC for converting only segmental features, such as mel-cepstrum and aperiodic components, of asource singer into those of a different target singer. The converted singing voice samples essentially have the segmental features of the target singer and the prosodic features, such as F0
patterns, power patterns, and duration, of the source singer.

4.2. Experimental results
Figure 1 shows the correlation between the perceptual age ofnatural singing voices and the actual age of the singer. Eachpoint shows the actual age of one singer and the average of theperceptual ages over all different songs sung by the same singer.
The correlation coefficient is 0.79. These results show quite
high correlation between the perceptual age and the actual age.
Table 2 shows average values and standard deviations ofdifferences between perceptual age of natural singing voices
and each type of intra-singer synthesized singing voice: analysis/synthesis (w/ AC), analysis/synthesis (w/o AC) and theintra-singer SVC. The table also shows correlation coefficients
between the perceptual age of natural and synthesized voices.
From the results, we can see that in analysis/synthesis (w/ AC),the perceptual age difference is small and the correlation coefficient is very high. Therefore, distortion caused by analysis/synthesis processing does not affect the perceptual age. It
can be observed from analysis/synthesis (w/o AC) that this re-

4. Experimental evaluation
4.1. Experimental conditions
In our experiments, we first investigated the correspondence
between the perceptual age and the actual age of the singer.
As test stimuli, we used all singing voices in the AIST humming database [21] consisting of singing voices of songs withJapanese lyrics sung by Japanese male and female amateur
singers in their 20s, 30s, 40s, and 50s. The total number ofthe singers was 75. Each singer sang 25 songs. The length of1059

Table 2: Differences of the perceptual age between natural singing voices and each type of the synthesized singing voices.
Methods
Average Standard deviation Correlation coefficient
Analysis/synthesis (w/ AC)
0.77
3.57
0.96
Analysis/synthesis (w/o AC)
0.44
3.58
0.96
Intra-singer SVC
-0.50
7.25
0.85

60

Target singers in their 20s (female
Target singers in their 30s (female
Target singers in their 40s (female
Target singers in their 50s (female

Female singer
Male singer
Regression line

Perceptual age of singing voices
generated by SVC

Perceptual age of singers

70

50
40
30
20
10
10

20

30

40

50

60

70

Actual age of singers
Figure 1: Correlation between singer’s actual age and perceptual age.

, male, male, male, male))))60
Regression line

55
50
45
40
35
30
25
20
20 25 30 35 40 45 50 55 60

Perceptual age of source singers
in intra-singer SVC

sult does not change even if not using aperiodic components.
Therefore, aperiodic components do not affect the perceptual
age of singing voices. On the other hand, intra-singer SVC
causes slightly larger differences between natural singing voices
and the synthesized singing voices. Therefore, some acoustic
cues to the perceptual age are removed through the statisticalconversion processing. Nevertheless, the perceptual age differences are relatively small, and therefore, it is likely that important acoustic cues to the perceptual age are still kept in theconverted acoustic features.
Figures 2 and 3 show a comparison between the perceptual
age of singing voices generated by SVC and intra-singer SVC.
In each figure, the vertical axis shows the perceptual age of converted singing voices by SVC (prosodic features: source singer,
segmental features: target singer). The horizontal axis in Fig. 2
shows the perceptual age of singing voices generated by intrasinger SVC (prosodic features: source singer, segmental features: source singer) and that in Fig. 3 shows the perceptual ageof singing voices generated by intra-singer SVC (prosodic features: target singer, segmental features: target singer). Therefore, if the prosodic features more strongly affect the perceptual
age than the segmental features, a higher correlation will be observed in Fig. 2. If the segmental features more strongly affectthe perceptual age than the prosodic features, a higher correlation will be observed in Fig. 3 than in Fig. 2. These figuresdemonstrate that 1) the segmental features affect the perceptual
age but the effects are limited as shown in positive but weakcorrelation in Fig. 3 and 2) the prosodic features have a larger
effect on the perceptual age than the segmental features.

Figure 2: Correlation of perceptual age between singing voices
generated by the intra-singer SVC and the SVC if setting horizontal axis to the perceptual age of the source singers.

Perceptual age of singing voices
generated by SVC

Source singers in their 20s (female
Source singers in their 30s (female
Source singers in their 40s (female
Source singers in their 50s (female, male, male, male, male))))60
Regression line

55
50
45
40
35
30
25
20
20 25 30 35 40 45 50 55 60

Perceptual age of target singers
in intra-singer SVC

Figure 3: Correlation of perceptual age between singing voices
generated by the intra-singer SVC and the SVC if setting horizontal axis to the perceptual age of the target singers.

prosodic features more strongly affect the perceptual age than
the segmental features. We plan to further study a conversion
technique for controlling the perceptual age of singing voices.

5. Conclusions
In this paper, we have investigated the acoustic features that
affect the perceptual age of singing voices. To factorize the effect of several acoustic features on the perceptual age of singing
voices, several types of synthetic singing voices were constructed and evaluated. The experimental results have demonstrated that 1) statistical voice conversion processing has only asmall effect on the perceptual age of singing voices and 2) the

6. Acknowledgements
Part of this work was supported by JSPS KAKENHI Grant
Number 22680016 and by the JST OngaCREST project.

1060

7. References[20] H. Kawahara, I. Masuda-Katsuse, and A. Cheveigne, “Restructuring speech representations using a pitch-adaptive time-frequency
smoothing and an instantaneous-frequency-based f0 extraction:
Possible role of a repetitive structure in sounds,” Speech Communication, vol. 27, no. 3-4, pp. 187–207, Apr. 1999.

[1] H. Kawahara and M. Morise, “Temporally variable multi-aspect
auditory morphing enabling extrapolation without objective andperceptual breakdown,” Proc. ICASSP, pp. 5389–5392, Mar.
2012.

[21] M. Goto and T. Nishimura, “AIST humming database: Music
database for singing research,” IPSJ SIG Notes (Technical Report)(Japanese edition), vol. 2005-MUS-61-2, pp. 7–12, Aug. 2005.

[2] Y. Stylianou, O. Cappé, and E. Moulines, “Continuous probabilistic transform for voice conversion,” IEEE Trans. SAP, vol. 6,
no. 2, pp. 131–142, Mar. 1998.
[3] T. Toda, A. W. Black, and K. Tokuda, “Voice conversion based on
maximum likelihood estimation of spectral parameter trajectory,”
IEEE Trans. ASLP, vol. 15, no. 8, pp. 2222–2235, Nov. 2007.
[4] F. Villavicencio and J. Bonada, “Applying voice conversion to
concatenative singing-voice synthesis,” Proc. INTERSPEECH,
pp. 2162–2165, Sept. 2010.
[5] Y. Kawakami, H. Banno, and F. Itakura, “GMM voice conversionof singing voice using vocal tract area function,” IEICE technicalreport. Speech (Japanese edition), vol. 110, no. 297, pp. 71–76,
Nov. 2010.
[6] T. Toda, Y. Ohtani, and K. Shikano, “One-to-many and many-toone voice conversion based on eigenvoices,” Proc. ICASSP, pp.
1249–1252, Apr. 2007.
[7] H. Doi, T. Toda, T. Nakano, M. Goto, and S. Nakamura, “Singing
voice conversion method based on many-to-many eigenvoice conversion and training data generation using a singing-to-singing
synthesis system,” Proc. APSIPA ASC, Nov. 2012.
[8] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Many-tomany eigenvoice conversion with reference voice,” Proc. INTERSPEECH, pp. 1623–1626, Sept. 2009.
[9] H. Zen, K. Tokuda, and A. W. Black, “Statistical parametricspeech synthesis,” Speech Communication, vol. 51, no. 11, pp.
1039–1064, Nov. 2009.
[10] T. Nose, J. Yamagishi, T. Masuko, and T. Kobayashi, “A stylecontrol technique for HMM-based expressive speech synthesis(speech and hearing),” IEICE transactions on information andsystems, vol. 90, no. 9, pp. 1406–1413, Sep. 2007.
[11] M. Tachibana, T. Nose, J. Yamagishi, and T. Kobayashi, “A technique for controlling voice quality of synthetic speech using multiple regression HSMM,” Proc. INTERSPEECH, pp. 2438–2441,
Sept. 2006.
[12] K. Ohta, T. Toda, Y. Ohtani, H. Saruwatari, and K. Shikano,“Adaptive voice-quality control based on one-to-many eigenvoice
conversion,” Proc. INTERSPEECH, pp. 2158–2161, Sept. 2010.
[13] H. Kasuya, H. Yoshida, S. Ebihara, and H. Mori, “Longitudinal changes of selected voice source parameters,” Proc. INTERSPEECH, pp. 2570–2573, Sept. 2010.
[14] N. Minematsu, M. Sekiguchi, and K. Hirose, “Automatic estimation of one’s age with his/her speech based upon acoustic modeling techniques of speakers,” Proc. ICASSP, pp. 137–140, May.
2002.
[15] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura, “Speech parameter generation algorithms for HMM-based
speech synthesis,” Proc. ICASSP, pp. 1315–1318, June 2000.
[16] H. Kawahara, J. Estill, and O. Fujimura, “Aperiodicity extraction
and control using mixed mode excitation and group delay manipulation for a high quality speech analysis, modification and systemstraight,” Proc. MAVEBA, Sept. 2001.
[17] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Maximum
likelihood voice conversion based on GMM with STRAIGHT
mixed excitation,” Proc. INTERSPEECH, pp. 2266–2269, Sept.
2006.
[18] T. Muramatsu, Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano,“Low-delay voice conversion based on maximum likelihood estimation of spectral parameter trajectory,” Proc. INTERSPEECH,
pp. 1076–1079, Sept. 2008.
[19] T. Toda, T. Muramatsu, and H. Banno, “Implementation of computationally efficient real-time voice conversion,” Proc. INTERSPEECH, Sept. 2012.

1061