Proceedings of the SMC 2009 - 6th Sound and Music Computing Conference, 23-25 July 2009, Porto - Portugal

VOCALISTENER: A SINGING-TO-SINGING SYNTHESIS SYSTEMBASED ON ITERATIVE PARAMETER ESTIMATION
Tomoyasu Nakano
Masataka Goto
National Institute of Advanced Industrial Science and Technology (AIST), Japan{t.nakano, m.goto} [at] aist.go.jp
ABSTRACTThis paper presents a singing synthesis system, VocaListener, that automatically estimates parameters for singing
synthesis from a user’s singing voice with the help of song
lyrics. Although there is a method to estimate singing synthesis parameters of pitch (F0 ) and dynamics (power) froma singing voice, it does not adapt to different singing synthesis conditions (e.g., different singing synthesis systems andtheir singer databases) or singing skill/style modifications.
To deal with different conditions, VocaListener repeatedly
updates singing synthesis parameters so that the synthesized singing can more closely mimic the user’s singing.
Moreover, VocaListener has functions to help modify theuser’s singing by correcting off-pitch phrases or changing
vibrato. In an experimental evaluation under two different
singing synthesis conditions, our system achieved synthesized singing that closely mimicked the user’s singing.
1 INTRODUCTION
Many end users have started to use commercial singing synthesis systems to produce music and the number of listeners
who enjoy synthesized singing is increasing. In fact, over
one hundred thousand copies of popular software packages
based on Vocaloid2 [1] have been sold and various compact
discs that include synthesized vocal tracks have appearedon popular music charts in Japan. Singing synthesis systems are used not only for creating original vocal tracks, butalso for enjoying collaborative creations and communications via content-sharing services on the Web [2, 3]. In lightof the growing importance of singing synthesis, the aim of
this study is to develop a system that helps a user synthesize natural and expressive singing voices more easily andefficiently. Moreover, by synthesizing high-quality humanlike singing voices, we aim at discovering the mechanism of
human singing voice production and perception.
Much work has been done on singing synthesis. The
most popular approach for singing synthesis is lyrics-tosinging (text-to-singing) synthesis where a user provides
note-level score information of the melody with its lyrics
to synthesize a singing voice [1, 4, 5]. To improve natuSMC 2009, July 23-25, Porto, Portugal
Copyrights remain with the authors

Page 343

ralness and provide original expressions, some systems [1]enable a user to adjust singing synthesis parameters such as
pitch (F0 ) and dynamics (power). The manual parameter
adjustment, however, is not easy and requires considerable
time and effort. Another approach is speech-to-singing synthesis where a speaking voice reading the lyrics of a song isconverted into a singing voice by controlling acoustic features [6]. This approach is interesting because a user can
synthesize singing voices having the user’s voice timbre, butvarious voice timbres cannot be used.
In this paper, we propose a new system named VocaListener that can estimate singing synthesis parameters (pitchand dynamics) by mimicking a user’s singing voice. Since anatural voice is provided by the user, the synthesized singing
voice mimicking it can be human-like and natural without
time-consuming manual adjustment. We named this approach singing-to-singing synthesis. Janer et al. [7] trieda similar approach and succeeded to some extent. Theirmethod analyzes acoustic feature values of the input user’s
singing and directly converts those values into the synthesis parameters. But their method is not robust with respect
to different singing synthesis conditions. For example, evenif we specify the same parameters, the synthesized results
always differ when we change to another singing synthesis
system or a different system’s singer database because of theresults’ nonlinearity. The ability to mimic a user’s singing
is therefore limited.
To overcome such limitations on robustness, VocaListener iteratively estimates singing synthesis parameters sothat after a certain number of iterations the synthesized
singing can become more similar to the user’s singing interms of pitch and dynamics. In short, VocaListener can
synthesize a singing voice while listening to its own generated voice through an original feedback-loop mechanism.
Figure 1 shows examples of synthesized voices under two
different conditions (different singer databases). With theprevious approach [7], there were differences in pitch (F0 )and dynamics (power). On the other hand, such differences
are minimal with VocaListener.
Moreover, VocaListener supports a highly-accurate
lyrics-to-singing synchronization function. Given the user’s
singing and the corresponding lyrics without any score
information, VocaListener synchronizes them automatically to determine each musical note that corresponds toa phoneme of the lyrics. We therefore developed an
originally-adapted/trained acoustic model for singing syn-

Proceedings of the SMC 2009 - 6th Sound and Music Computing Conference, 23-25 July 2009, Porto - Portugal

Target singing
and song lyrics

Acoustic
featuresAta chi do ma ru to ki ma ta fu to fu ri ka e ru
Analyze the target singing
phonetic alignment
t

ch

d m

a

i o a u o i

r t k

B

m

t

a

F0 and vibrato sections

f

t

f

a u o

r

u

k

i

r

a

e

u

power

VocaListener based on iterative parameter estimation

C

Apply VocaListener-plus (to adjust the singing skill/style)

Synthesized
singingta chi do ma ru to ki

pitchma ta fu to

fu ri

ka

e

ru

dynamics

Singing synthesis (2 singing synthesis conditions)

Condition II

Condition I

Analyze the synthesized singing (final outputs)
F0
power
Acoustic
featuresUpdate the parameters

Singingsynthesis
parameters

Set synthesis parameters to initial values
lyrics alignment

are taken as the system input (!
A ). Using this input, thesystem automatically synchronizes the lyrics with the target
singing to generate note-level score information, estimates
the fundamental frequency (F0 ) and the power of the target
singing, and detects vibrato sections that are used just for the
VocaListener-plus (!
B ). Errors in the lyrics synchronization
can be manually corrected through simple interaction. The
system then iteratively estimates the parameters through theC ).
VocaListener-core, and synthesizes the singing voice (!
The user can also adjust the singing skill/style (e.g., vibrato
extent and F0 contour) through the VocaListener-plus.
2.1 VocaListener-front-end: analysis and synthesis
The VocaListener-front-end consists of singing analysis andsinging synthesis. Throughout this paper, singing samples
are monaural recordings of solo vocal digitized at 16 bit /44.1 kHz.
2.1.1 singing analysis

Outputs can closely mimic the target singing
because of iterative parameter estimation
Previous approachD

Map acoustic feature values directly into synthesis parameters
lyrics alignment
Singingsynthesis
parameters

Synthesized
singingta chi do ma ru to ki

pitchma ta fu to

fu

ri

ka e

ru

dynamics

Singing synthesis (2 singing synthesis conditions)

Condition II

Condition I

Outputs cannot mimic the target singing wellbecause different conditions cause different synthesized results
F0

power

Acoustic
features

Figure 1. Overview of VocaListener and problems of a previous
approach by Janer et al. [7].

chronization. Although synchronization errors with thismodel are rare, we also provide an interface that lets a user
easily correct such errors just by pointing them out. In addition, VocaListener also supports a function to improve synthesized singing as if the user’s singing skill were improved.

The system estimates the fundamental frequency (F0 ), thepower, and the onset time and duration of each musical note.
Since the analysis frame is shifted by 441 samples (10 ms),
the discrete time step (1 frame-time) is 10 ms. This paper
uses time t for the time measured in frame-time units.
In VocaListener, these features are estimated as follows:
Fundamental frequency: F0 (t) is estimated using
SWIPE [8]. Hereafter, unless otherwise stated, F0 (t)are log-scale frequency values (real numbers) in relation
to the MIDI note number (a semitone is 1, and middle
C corresponds to 60).
Power: P ow(t) is estimated by applying a Hanning window whose length is 2048 samples (about 46 ms).
Onset time and duration: To estimate the onset time andduration of each musical note, the system synchronizes
the phoneme-level pronunciation of the lyrics with thetarget singing. This synchronization is called phoneticalignment and is estimated through Viterbi alignment
with a phoneme-level hidden Markov model (monophone HMM). The pronunciation is estimated by using
a Japanese language morphological analyzer [9].
2.1.2 singing synthesis

2 PARAMETER ESTIMATION SYSTEM FORSINGING SYNTHESIS: VOCALISTENER
VocaListener consists of three components, the
VocaListener-front-end for singing analysis and synthesis, the VocaListener-core to estimate the parameters forsinging synthesis, and the VocaListener-plus to adjust thesinging skill/style of the synthesized singing.
Figure 1 shows an overview of the VocaListener system.
The user’s singing voice (i.e., target singing) and the lyrics1
1 In our current implementation, Japanese lyrics spelled in a mixture ofJapanese phonetic characters and Chinese characters are mainly supported.
English lyrics can also be easily supported because the underlying ideas of
VocaListener are universal and language-independent.

Page 344

In our current implementation, the system estimates parameters for commercial singing synthesis software based on
Yamaha’s Vocaloid or Vocaloid2 technology [1]. For example, we use software named Hatsune Miku (referred to asCV01) and Kagamine Rin (referred to as CV02) [10] forsynthesizing Japanese female singing. Since all parameters
are estimated every 10 ms, they are linearly interpolated atevery 1 ms to improve the synthesized quality, and are fed
via a VSTi plug-in (Vocaloid Playback VST Instrument).
2.2 VocaListener-plus: adjusting singing skill/style
To extend the flexibility, the VocaListener-plus provides
functions, pitch change and style modification, which can

2.2.1 Pitch change
We propose pitch transposition and off-pitch correction to
overcome the limitations of the user’s singing skill and pitch
range. The pitch transposition function changes the target
F0 (t) just by adding an offset value for transposition during
the whole section or a partial section.
The off-pitch correction function automatically corrects
off-pitch phrases by adjusting the target F0 (t) according to
an offset of Fd (0 ≤ Fd < 1) estimated for each voiced
section. The off-pitch amount Fd is estimated by fitting
a semitone-width grid to F0 (t). The grid is defined as a
comb-filter-like function where Gaussian distributions arealigned at one semitone intervals. Just for this fitting, F0 (t)is temporarily smoothed by using an FIR lowpass filter
with a 3-Hz cutoff frequency2 to suppress F0 fluctuations(overshoot, vibrato, preparation, and fine fluctuation) of thesinging voice [11, 12]. Last, the most fitted offset Fd is used
to adjust F0 (t) to its nearest correct pitch.

70
69
68
67
66
65
64
63
62

Suppress vibrato extent
Correct off-pitch phrase well0

1

In this paper, vibrato adjustment and singing smoothing
are proposed to emphasize or suppress the F0 fluctuations.
Since the F0 fluctuations are important factors to characterize human singing [11, 12], a user can change the impression of singing. The F0 (t) and P ow(t) of the target singing
are adjusted by interpolating or extrapolating between theoriginal values (F0 (t) and P ow(t)) and their smoothed values obtained by using an FIR lowpass filter. A user canseparately adjust vibrato sections and other sections. The
vibrato sections are detected by using the vibrato detection
method [13].

2

3

4

5

6

time [s]ta chi do ma ru to ki

Target singing and song lyrics

time

VocaListener-core
Lyrics alignment(1) Adjustment of voiced sections iteratively(i) Set vowel alignment to initial valuesA(ii) Connect two adjacent notes if they are in a voiced section(iii) Stretch note onset and offset(iv) Estimate the note number and synthesize the singing
ch
r
t
t
d
m
k
a
i
o
a
u
o
i
ta
ch i d o
ma
ru to
ki
(i)
(ii)

ta

ch i

do

ma

ru to

(iii)

ta

ch i

do

ma

ru

ki

to

ki

voiced / unvoiced sections (initial output)

Target singing

Synthesized singing(2) Repairing boundary errorta

2.2.2 Style modification

Adjusted by VocaListener-plus

Target singing

Figure 2. Example of F0 (t) adjusted by VocaListener-plus.

Updatemodify the value of the estimated acoustic features of thetarget singing. The user can select whether to use thesefunctions based on personal preference. Figure 2 shows anexample of using these functions.

F0 [semitone]

Proceedings of the SMC 2009 - 6th Sound and Music Computing Conference, 23-25 July 2009, Porto - Portugal

ch i

Select new
boundary
candidates

do

ma

ru

do

ma

ru

Pointed out by user
to
ki
to

do

ma

ru to

do

ma

ru

to

voicedtime

B

Estimate
pitch parameters
Synthesize the singing
Compute distanceOne candidate with minimum distance is presented to user

If the correct boundary cannot be obtained,
the user can point to it again or correct manually

Pitch parameter estimation- Note number
ta
64

ch i
63

do
64

ma
63

ru

64

to

66

- Pitch Bend (PIT) and Pitch Bend Sensitivity (PBS)
PIT

ki

68

time

Dynamics parameter estimation- Dynamics (DYN)
DYN

time

2.3 VocaListener-core: estimating the parameters
Figure 3 shows the estimation process for VocaListenercore. After acoustic features of the target singing (modified by VocaListener-plus, if necessary) are estimated, thesefeatures are converted into synthesis parameters that arethen fed to the singing synthesis software. The synthesized singing is then analyzed and compared with the target
singing. Until the synthesized singing is sufficiently close to
the target singing, the system repeats the parameter update
and its synthesis.
2.3.1 Parameters for singing synthesis
The system estimates parameters for pitch, dynamics, andlyrics alignment (Table 1). The pitch parameters consist of
MIDI note number (Note#)3 , pitch bend (PIT), and pitch
2 We avoid unnatural smoothing by ignoring silent sections and leaps of
F0 transitions wider than a 1.8-semitone threshold.
3 For synthesis, each mora of Japanese pronunciation is mapped into amusical note, where the mora representation can be classified into three

Page 345

Synthesized singing

time

Figure 3. Overview of the parameter estimation procedure,
VocaListener-core.

bend sensitivity (PBS), and the dynamics parameter is dynamics (DYN). For the pitch (F0 ), the fractional portion (PIT)
is separated from the integer portion (Note#). PIT represents
a relative decimal deviation from the corresponding integer
note number (Note#), and PBS specifies the range (magnitude) of its deviation. The results of the lyrics alignment are
represented by the note onset (onset time) and its duration.
These MIDI-based parameters can be considered typicaland common, not specific to the Vocaloid software. A setof these parameters, PIT, PBS, and DYN, are iteratively estimated after being initialized to 0, 1, and 64, respectively.
types: “V”, “CV”, and “N”. “V” denotes vowel (a, i, ...), “C” denotes
consonant (t, ch, ...), and “N” denotes syllabic nasal (n).

Table 1. Relation between singing synthesis parameters and acoustic features.
Acoustic features
F0
Power
Phoneticalignment

Synthesis parameters
Pitch
Note#, PIT, and PBS
Dynamics
DYN
Lyrics
Note onsetalignment
Note duration

F0 [semitone]

Proceedings of the SMC 2009 - 6th Sound and Music Computing Conference, 23-25 July 2009, Porto - Portugal

68

Target singing

66

MIDI note number

64
62
0

Lyrics

0.5
ta

1

ch i

do

ma

1.5
ru

to

2

time [s]ki

Figure 4. F0 of the target singing and estimated note numbers.

2.3.2 Lyrics alignment estimation with error repairing

Step (i) Given the phonetic alignment of the automatic synchronization, the note onset and duration are initialized by using
its vowel.
Step (ii) If two adjacent notes are not connected but their sections
are judged to be a single voiced section, the duration of theformer note is extended to the onset of the latter note sothat they can be connected. This eliminates a small gap andimproves the naturalness of the synthesized singing.
Step (iii) By comparing voiced sections of the target and synthesized singing, the note onset and duration are adjusted sothat they become closer to those of the target.
Step (iv) Given the new alignment, the note number (Note#) is
estimated again and the singing is synthesized.

Although the automatic synchronization of song lyrics
with the target singing is accurate in general, there are sometimes a few boundary errors that degrade the synthesized
quality. We therefore propose an interface that lets a usercorrect each error just by pointing it out without manually
adjusting (specifying) the boundary. As shown in Figure
3!
B , other boundary candidates are shown on a screen sothat the user can simply choose the correct one by listening
to each one. Even if it is difficult for a user to specify the correct boundary from scratch, it is easy to choose the correct
candidate interactively. To generate candidates, the systemcomputes timbre fluctuation values of the target singing byusing ∆MFCCs, and several candidates with high fluctuation values are selected. The system then synthesizes eachcandidate and compares it with the target singing by using
MFCCs. The candidates are sorted and presented to the user
in the order of similarity to the target singing. If none of thecandidates are correct, the user can correct manually at theframe level.
2.3.3 Pitch parameter estimation
Given the results of lyrics alignment, the pitch parameters
are iteratively estimated so that the synthesized F0 can become closer to the target F0 . After the note number of each

Page 346

Target singing
Synthesized singing
A

Power

Even if the same note onset and its duration (lyrics alignment) are given to different singing synthesis systems (suchas Vocaloid and Vocaloid2) or different singer databases(such as CV01 and CV02), the note onset and note duration often differ in the synthesized singing because of theirnonlinearity (caused by their internal waveform concatenation mechanism). We therefore have to adjust (update) the
lyrics alignment iteratively so that each voiced section of thesynthesized singing can be the same as the original voiced
section of the target singing. As shown in Figure 3!
A , thelast two steps (iii) and (iv) in the following four steps arerepeated:0

Lyrics

0.5
ta

1

ch i

do

DYN = 127
DYN = 96
DYN = 64
DYN = 32
1.5

ma

ru

to

2

time [s]ki

Figure 5. Power of the target singing and power of the singing
synthesized with four different dynamics.

note is estimated, PIT and PBS are repeatedly updated byminimizing a distance between the target F0 and the synthesized F0 .
The note number N!ote# for each note is estimatedby%Note# = argmax
n

"
t

$
#
(n − F0 (t))2
exp −
,
2σ 2

(1)

where n denotes a note number candidate, is set to 0.33,
and t is 0 at the note onset and continues for its duration.
Figure 4 shows an example of F0 and its estimated notenumbers.
The PIT and PBS are then estimated by repeating the following steps, where i is the number of updates (iterations),
F 0org (t) denotes F0 of the target singing, and PIT and PBS
are represented by PIT(i) (t) and PBS(i) (t):Step 1) Obtain synthesized singing from the current parameters.
(i)

Step 2) Estimate F 0syn (t) that denotes F0 of the synthesized
singing.
Step 3) Update P b(i) (t) by&'
P b(i+1) (t) = P b(i) (t) + F 0org (t) − F 0(i)(2)
syn (t) ,where P b(i) (t) is a log-scale frequency computed from
PIT(i) (t) and PBS(i) (t).

Step 4) Obtain the updated PIT(i+1) (t) and PBS(i+1) (t) fromP b(n+1) (t) after minimizing PBS(i+1) (t). Since a smaller
PBS gives better resolution of the synthesized F0 , PBS
should be minimized at every iteration as long as PIT canrepresent the correct relative deviation.

2.3.4 Dynamics parameter estimation
Given the results of lyrics alignment and the pitch parameters, the dynamics parameter is iteratively estimated so thatthe synthesized power can be closer to the target power. Figure 5 shows the power of the target singing before normalization and the power of the singing synthesized with four
different dynamics. Since the power of the target singing
depends on recording conditions, it is important to mimic
the relative power after normalization that is determined soProceedings of the SMC 2009 - 6th Sound and Music Computing Conference, 23-25 July 2009, Porto - Portugal

Table 2. Dataset for experiments A and B and synthesis conditions. All of the song samples were sung by female singers.
Exp.
No.
A
A
B
B
B
B

Song
No.
No.07
No.16
No.07
No.16
No.54
No.55

Excerpted
sectionintro–verse–chorus
intro–verse–chorus
verse A
verse A
verse A
verse A

Length
[s]
103
100
6.0
7.0
8.9
6.5

Synthesis
conditions
CV01
CV02
CV01, CV02
CV01, CV02
CV01, CV02
CV01, CV02

that the normalized target power can be covered by the synthesized power with DYN = 127 (maximum value). However, because there are cases where the target power exceeds
A ), the synthethe limit of synthesis capability (e.g., Fig.5!
sized power cannot perfectly mimic the target. As a compromise, the normalization factor α is determined by minimizing an error defined as a square error between αP oworg (t)DYN=64
DYN=64and P owsyn(t), where P owsyn(t) denotes the synthesized power with DYN = 64.
The DYN is then estimated by repeating the following
steps, where P oworg (t) denotes the power of the target
singing:
Step 1) Obtain synthesized singing from the current parameters.
(i)

Step 2) Estimate P owsyn (t) that denotes the power of the synthesized singing.
Step 3) Update Db(i) (t) by&'(i)(t) , (3)Db(i+1) (t) = Db(i) (t) + αP oworg (t) − P owsyn
where Db(i) (t) is the actual power given by the current
DYN.
Step 4) Obtain the updated DYN from Db(i+1) (t) by using therelationship between the DYN and the actual power values. Before these iteration steps, this relationship should be
investigated once by synthesizing the current singing withfive DYN values (= 0, 32, 64, 96, 127). The relationshipfor each of the other DYN values is linearly interpolated.

3 EXPERIMENTAL EVALUATIONSThe VocaListener was tested in two experiments. Experiment A evaluated the number of times manual corrections
had to be made, and experiment B evaluated the performance of the iterative estimation under different conditions.
In these experiments, two singer databases, CV01 andCV02, were used with the default software settings except
for the note-level properties of “No Vibrato” and “0% Bend
Depth”. Unaccompanied song samples (solo vocal) weretaken from the RWC Music Database (Popular Music [14]),
and were used as the target singing as shown in Table 2.
For the automatic synchronization of the song lyrics inexperiment A, a speaker-independent HMM provided by
CSRC [15] for speech recognition was used as the basic
acoustic model for MFCCs, ∆MFCCs, and ∆power. The
HMM was adapted with singing voice samples by applying
MLLR-MAP [16]. As in cross validation where one song
sample is evaluated as the test data and the other samples
are used as the training data, we excluded the same singer
from the HMM adaptation data.

Page 347

3.1 Experiment A: interactive error repairing for lyricsalignment
To evaluate the lyrics alignment, experiment A used two female songs that were over 100 s in length. Table 3 shows thenumber of boundary errors that had to be repaired (pointedout) and the number of repairs needed to correct those errors4 . For example, among 128 musical notes for song
No.16, there were only three boundary errors that should
be manually pointed out on our interface, and two of thesewere pointed out twice. In other words, one error was corrected by choosing the first candidate, and the other two errors were corrected by choosing the second candidate. In
our experience with many songs, errors tend to occur around/w/ or /r/ (semivowel, liquid) and /m/ or /n/ (nasal sound).
3.2 Experiment B: iterative estimation experiment
Experiment B used four song excerpts sung by four female
singers. As shown in Table 2, each song was tested with two
conditions — i.e., two singer databases, CV01 and CV02.
Since the experiment focused on the performance of the iterative estimation for the pitch and dynamics, we used thehand-labeled lyrics alignment here. The results were evaluated by the mean error value defined by"
1 ! """(i)(4)"F 0org (t) − F 0syn (t)" ,
Tf t
#
$"
1 ! """(i)(i)
errpow ="20 log (αP oworg (t)) − 20 log P owsyn (t) " , (5)
Tp t
(i)

errf0 =

where Tf denotes the number of voiced frames, and Tp denotes the number of nonzero power frames.
Table 4 shows the mean error values after each iteration
for song No.07, where the “×n” column denotes the numberof iterations before synthesis and the “×0” column denotes
initial synthesis without any iteration. Starting from large
errors of initial synthesis (“×0”), the mean error values weremonotonically decreased after each iteration and the synthesized singing after the fourth iteration (“×4”) was mostsimilar to the target singing. The results for the other songs
also showed similar improvement as shown in Table 5. The“Previous approach” column in Tables 4 and 5 denotes theresults of mapping acoustic feature values directly into synthesis parameters (almost equivalent to [7]). The mean error
values after the fourth iteration were much smaller than theprevious approach. In fact, when we listened to those synthesized results, the synthesized results after the fourth iteration (“×4”) were clearly better than the synthesized results
without any iteration (“×0” and “Previous approach”).
3.3 Discussion
The results of experiment A show that our automatic synchronization (lyrics alignment) worked well. Even if there
were a few boundary errors (eight errors among 166 notes
in No.07 and three errors among 128 notes in No.16), they
4 This table does not show another type of error where the global phraseboundary was wrong. There were two such errors in No.16 and they could
also be corrected through simple interaction (just by moving roughly).

Proceedings of the SMC 2009 - 6th Sound and Music Computing Conference, 23-25 July 2009, Porto - Portugal

Table 3. Number of boundary errors and number of repairs for correcting (pointing out) errors in experiment A.
Song
No.

Synthesis
conditions

Numberof notes

No.07
No.16

CV01
CV02

166
128

Number of boundary
errors after each repair
×0 ×1
×2 ×3
8
5
2
0
3
2
0
—

Table 4. Mean error values after each iteration for song No.07 in experiment B.
(i)Parameters

Synthesis
conditions

Pitch
Pitch
Dynamics
Dynamics

CV01
CV02
CV01
CV02

could be easily corrected by choosing from the top threecandidates. We thus confirmed that our interface for correcting boundary errors was easy-to-use and efficient. Moreover, we recently developed an original acoustic model thatwas trained from scratch with singing voices including awide range of vocal timbres and singing styles. Although
we did not use this high-performance model in the above
experiments, our preliminary evaluation results suggest that
more accurate synchronization can be achieved.
The results of experiment B show that iterative updates
were an effective way to mimic the target singing under various conditions. In addition, we tried to estimate the parameters for CV01/CV02 using song samples synthesized with
CV01 as the target singing, and confirmed that the estimatedparameters for CV01 were almost same with the original
parameters and the synthesized singing with CV01/CV02
sufficiently mimicked the target singing. VocaListener canthus be used not only for mimicking singing by human, butalso for re-estimating the parameters under different synthesis conditions without time-consuming manual adjustment.
4 CONCLUSION
We have described a singing-to-singing synthesis system,
VocaListener, that automatically estimates parameters forsinging synthesis by mimicking a user’s singing. The experimental results indicate that the system effectively mimics
target singing with error values decreasing with the numberof iterative updates. Although Japanese lyrics are currently
supported in our implementation, our approach can be utilized for any other language.
In our experience of synthesizing various songs with VocaListener using seven different singer databases on two different singing synthesis systems (Vocaloid and Vocaloid2),
we found the synthesized quality was high and stable5 . One
benefit of VocaListener is that a user does not need to perform time-consuming manual adjustment even if the singer
database changes. Before VocaListener, this problem was
widely recognized and many users had to repeatedly adjust
parameters. With VocaListener, once a user synthesizes asong based on the target singing (even synthesized singing
the user has adjusted in the past), its vocal timbre can be easily changed just by switching a singer database on our interface. Since this ability is very useful for end users, we namethis meta-framework a Meta-Singing Synthesis System. We
hope that a future singing synthesis framework will support
this promising idea, thus expediting wider use of singing
5 A demonstration video including examples of synthesized singing isavailable at http://staff.aist.go.jp/t.nakano/VocaListener/.

Page 348(i)

Mean error values (errf0 [semitone] and errpow [dB])Previous
VocaListener
approach
×0
×1
×2
×3
×4
0.217
0.386
0.091
0.058 0.042
0.034
0.198
0.352
0.074
0.041 0.029
0.024
13.65
11.22
4.128
3.617 3.472
3.414
14.17
15.26
6.944
6.382 6.245
6.171

Table 5. Minimum and maximum error values for all four songs
in experiment B.
Parameters
Pitch
Dynamics

Mean error values (min−max)
Previous
VocaListener
approach×0×40.168−0.369
0.352−1.029 0.019−0.107
9.545−15.45
10.46−19.04 1.676−6.560

synthesis systems to produce music.
5 ACKNOWLEDGEMENTS
We thank Jun Ogata (AIST), Takeshi Saitou (CREST/AIST), andHiromasa Fujihara (AIST) for their valuable discussions. This research was supported in part by CrestMuse, CREST, JST.

6 REFERENCES[1] Kenmochi, H. et al. “VOCALOID – Commercial Singing Synthesizer based on Sample Concatenation,” Proc. INTERSPEECH 2007,
pp.4011–4010, 2007.
[2] Hamasaki, M. et al. “Network Analysis of Massively Collaborative
Creation of Multimedia Contents: Case Study of Hatsune Miku Videos
on Nico Nico Douga,” Proc. uxTV’ 08, pp.165–168, 2008.
[3] Cabinet Office, Government of Japan. “Virtual Idol,” Highlighting
JAPAN through images, Vol.2, No.11, pp.24–25, 2009.
http://www.gov-online.go.jp/pdf/hlj img/vol 0020et/24-25.pdf[4] Bonada, J. et al. “Synthesis of the Singing Voice by Performance
Sampling and Spectral Models,” IEEE Signal Processing Magazine,
Vol.24, Iss.2, pp.67–79, 2007.
[5] Saino K. et al. “HMM-based singing voice synthesis system, ” Proc.
ICSLP06, pp.1141–1144, 2006.
[6] Saitou, T. et al. “Speech-To-Singing Synthesis: Converting Speaking
Voices to Singing Voices by Controlling Acoustic Features Unique toSinging Voices,” Proc. WASPAA2007, pp.215–218, 2007.
[7] Janer, J. et al.: “Performance-Driven Control for Sample-BasedSinging Voice Synthesis,” Proc. DAFx-06, pp.42–44, 2006.
[8] Camacho, A. “SWIPE: A Sawtooth Waveform Inspired Pitch Estimator for Speech and Music,” Ph.D. Thesis, University of Florida, 116 p.,
2007.
[9] Kudo, T. “MeCab: Yet Another Part-of-Speech and Morphological
Analyzer”. http://mecab.sourceforge.net/[10] Crypton Future Media. “What is the HATSUNE MIKU movement?,”
http://www.crypton.co.jp/download/pdf/info miku e.pdf[11] Saitou, T. et al. “Development of an F0 control Model Based on F0
Dynamic Characteristics for Singing-Voice Synthesis,” Speech Communication, Vol.46, pp.405-417, 2005.
[12] Mori, H. et al. “F0 Dynamics in Singing: Evidence from the Dataof a Baritone Singer,” IEICE Trans. Inf. & Syst., Vol.E87-D, No.5,
pp.1086–1092, 2004.
[13] Nakano, T. et al. “An Automatic Singing Skill Evaluation Method for
Unknown Melodies Using Pitch Interval Accuracy and Vibrato Features,” Proc. ICSLP 2006, pp.1706–1709, 2006.
[14] Goto, M. et al. “RWC Music Database: Popular, Classical, and Jazz
Music Databases,” Proc. ISMIR 2002, pp.287–288, 2002.
[15] Lee, A. et al. “Continuous Speech Recognition Consortium – An Open
Repository for CSR Tools and Models –,” Proc. LREC2002, pp.1438–
1441, 2002.
[16] Digalakis, V.V. et al.: “Speaker Adaptation Using Combined Transformation and Bayesian Methods,” IEEE Transactions on Speech andAudio Processing, Vol.4, No.4, pp.294–300, 1996.