Automatic synchronization between lyrics and music CD recordings
based on Viterbi alignment of segregated vocal signals
Hiromasa Fujihara,† Masataka Goto,‡ Jun Ogata,‡
Kazunori Komatani,† Tetsuya Ogata,† and Hiroshi G. Okuno††

Dept. of Intelligence Science and Technology
Graduate School of Informatics, Kyoto University
Sakyo-ku, Kyoto 606-8501, Japan{fujihara, komatani, ogata, okuno}@kuis.kyoto-u.ac.jp
Abstract
This paper describes a system that can automaticallysynchronize between polyphonic musical audio signals andcorresponding lyrics. Although there were methods that cansynchronize between monophonic speech signals and corresponding text transcriptions by using Viterbi alignment
techniques, they cannot be applied to vocals in CD recordings because accompaniment sounds often overlap with vocals. To align lyrics with such vocals, we therefore developed three methods: a method for segregating vocals frompolyphonic sound mixtures, a method for detecting vocal
sections, and a method for adapting a speech-recognizer
phone model to segregated vocal signals. Experimental results for 10 Japanese popular-music songs showed that oursystem can synchronize between music and lyrics with satisfactory accuracy for 8 songs.

1. Introduction
A vocal track and its lyrics play an important role inmany musical genres, especially in popular music. To represent the theme and story of the song, they are essential
elements that create an impression of the song. When asong is heard, for example, most people listen to the vocal melody and follow the lyrics. This is why music videos
often display synchronized lyrics as a caption, helping theaudiences enjoy the music.
In this paper we propose an automatic synchronization
system for polyphonic audio signals of songs and theirlyrics. This system can automatically estimate the temporal relationship (alignment) between audio signals and thecorresponding lyrics. This approach is different from direct
lyrics recognition like speech recognition and takes advantage of the vast amount of lyrics embedded in the web. Our
system has various applications, such as automatic genera-

Proceedings of the Eighth IEEE International Symposium on Multimedia (ISM'06)
0-7695-2746-9/06 $20.00 © 2006‡

National Institute of Advanced Industrial
Science and Technology (AIST)
Tsukuba, Ibaraki 305-8568, Japan{m.goto, jun.ogata}@aist.go.jp

tion of music video captions and a music playback interface
that can directly access to specific words or passages of interest.

Wang et al. [19] have worked on a similar system. They
have integrated higher structural information (such as beattracking and chorus detection) and lower level lyrics alignment. Their lower level lyrics alignment method uses only
the duration of each phoneme as a cue. However, thismethod is not consistently effective because the durations ofuttered phonemes differ based on location, even though they
are the same phonemes. The method also requires many assumptions about the structure and meter of the song in orderto obtain higher structural information. Other related studies have focused on lyrics recognition[8, 16, 7]. They use aspeech recognizer for lyrics recognition. These studies presume pure monophonic singing voices without accompaniment, posing additional difficulties for practical use withmusical audio signals like CD recordings.

Because current speech recognition techniques are incapable of automatically synchronizing lyrics with music including accompaniments, we developed three methods: amethod for segregating vocal (singing) signals from polyphonic audio signals, a method for detecting sections including vocal signals, and a method for adapting a phonemodel of speech recognizers to segregated vocal signals.

The rest of this paper is organized as follows. In thenext section, we describe an overview of our system for automatic synchronization between music and lyrics. From
Section 3 to Section 5, we describe our system in detail. In
Section 6, we describe our experiments and present the results. In Section 7, we draw conclusions and discuss future
directions.

2. Automatic Synchronization between Musicand Lyrics

Input (Polyphonic audio signals)

Given musical audio signals and the corresponding
lyrics, our system can locate the start and end times for eachphrase of the lyrics by automatic synchronization between
music and lyrics. The system deals with real-world musical audio signals such as popular music CD recordings that
contain a singer’s vocal track and various accompanimentsounds. We assume that the main vocal part is sung bya single predominant singer (except for choruses), but do
not assume anything about the number and kind of soundsources in the accompaniment sounds.
To solve this problem, the basic idea is to use the Viterbialignment (forced alignment) technique that is often usedin automatic speech recognition. This technique, however,does not work well when there are accompaniment sounds
that are performed together with a vocal and when there
are interlude sections during which the vocal is not performed. We therefore first extract and resynthesize the harmonic structure of the vocal melody and obtain the vocal
signals by using an accompaniment sound reduction methodthat we proposed previously [4]. Then, we detect the vocal sections (regions) from segregated vocal signals by using a vocal activities detection method based on a Hidden
Markov Model (HMM). Finally, we align the lyrics with thesegregated vocal audio signals by using a Viterbi alignment
technique. We also propose a method for adapting a phonemodel to the segregated vocal signals of the target singer.

1. F0 Estimation

3. Accompaniment Sound Reduction
To extract a vocal feature representing the phonetic information of a singing voice from polyphonic audio signals,
it is necessary to reduce the influence of accompanimentsounds. We therefore resynthesize vocal signals from theharmonic structure of the melody line by the following threeparts:
1. Estimating the fundamental frequency (F0) of the
melody line (vocal) in CD recordings by using a
predominant-F0 estimation method called PreFEst [5].
2. Extracting the harmonic structure corresponding to theF0 of the melody line.
3. Resynthesizing the audio signal (waveform) corresponding to the melody line using a sinusoidal synthesis.
Thus, we obtain a waveform corresponding only to the
melody line. Figure 1 shows an overview of the accompaniment sound reduction method. Note that the melody line
obtained with this method may contain instrumental (i.e.,
non-vocal) sounds in interlude sections as well as voices invocal sections, because the melody line here is defined as
the most predominant F0 in each frame [5]. It is therefore
necessary to detect the vocal sections by using the method
described in section 4.

Proceedings of the Eighth IEEE International Symposium on Multimedia (ISM'06)
0-7695-2746-9/06 $20.00 © 2006

2. Harmonic Structure
Extraction

3. Resynthesis
Output (Melody’s audio signals)
Figure 1. Accompaniment Sound Reduction.

3.1. F0 Estimation
We use Goto’s PreFEst [5] to estimate the F0 of the
melody line. PreFEst estimates the most predominant F0 in
frequency-range-limited sound mixtures. Since the melody
line tends to have the most predominant harmonic structurein middle- and high-frequency regions, we can estimate theF0s of the melody line by applying PreFEst with adequate
frequency-range limitation.
We describe a summary of PreFEst below. Hereafter, x is
the log-scale frequency denoted in units of cents (a musicalinterval measurement), and t is discrete time. Although acent originally represented tone interval (relative pitch), we3
use it as a unit of absolute pitch using 440 × 2 12 −5 Hz as acriterion, according to Goto [5]. The conversion from Hz tocent is expressed as follows:
fcent = 1200 log2

fHz
3

440 × 2 12 −5,(1)where fcent and fHz represent frequency in cents and Hz,
respectively.
(t)

Given the power spectrum Ψp (x), we first apply aband-pass filter (BPF) that is designed so that it covers
most of the dominant harmonics of typical melody lines.
The filtered frequency components can be represented as(t)
BP F (x)Ψp (x), where BP F (x) is the BPF’s frequency
response for the melody line. In this paper, we designed
the BPF according to Goto’s specifications [5]. To makethe application of statistical methods possible, we represent
each of the bandpass-filtered frequency components as aprobability density function (PDF), called an observed PDF,



(t)

pΨ (x):sl (k)(t)
BP F (x)Ψp (x)(t)
pΨ (x) =  ∞.
BP F (x)Ψ(t)
(x)dx
p
−∞

s(k) =

Fl

θ(t) = {w(t) (F )|Fl ≤ F ≤ Fh},(4)

where p(x|F ) is the PDF of the tone model for each F0, andFh and Fl are defined as the lower and upper limits of thepossible (allowable) F0 range, and w(t) (F ) is the weight ofa tone model that satisfies
 Fliw(t) (F )dF = 1.
(5)
Fhi

A tone model represents a typical harmonic structure and indicates where the harmonics of the F0 tend to occur. Then,we estimate w(t) (F ) using an EM algorithm and regard it as
the F0’s PDF. Finally, we track a dominant peak trajectoryof F0s from w(t) (F ) using multiple agent architecture.

3.2. Harmonic Structure Extraction
Based on the estimated F0, we extract the power of fundamental frequency component and harmonic components.
For each component, we allow r cent error and extract thepeak in the allowed area. The power Al and frequency Flof l th overtone (l = 1, . . . , L) can be represented asAl= argmax |S(F )|
F

=

r

r

(lF · (1 − 2 1200 ) ≤ F ≤ lF · (1 + 2 1200 )), (6)|S(Fl )|,
(7)

where S(F ) denotes the spectrum and F denotes the F0
estimated by PreFEst. In our experiments, we set r to 20.

3.3. Resynthesis
We resynthesize the audio signals of the melody linefrom the extracted harmonic structure using a sinusoidal
model [12]. We denote the frequency and the amplitude of l(t)(t)
th overtone at time t as Fl and Al , respectively. Changesof a phase are approximated using a quadratic function sothat a frequency changes linearly, and changes of amplitude
are approximated using linear function. Resynthesized audio signals, s(k), are expressed asθl (k)

=

(t+1)
(t)
− Fl ) 2
π(Fl

K

(t)

(t)

k + 2πFl k + θl,0 , (8)Proceedings of the Eighth IEEE International Symposium on Multimedia (ISM'06)
0-7695-2746-9/06 $20.00 © 2006

K


(t)
+ Al

sin (θl (k)) ,
(9)

(2)

Then, we consider each observed PDF to have been generated from a weighted-mixture model of the tone models of
all the possible F0s, which is represented as follows: Fhw(t) (F )p(x|F )dF(3)p(x|θ(t) ) =

Fl

=

(t+1)
(t) k
(Al
− Al )Lsl (k),(10)l=1where k represents a time in units of seconds and defines
time t as k = 0, K represents the duration between (t) and(t)(t + 1) in units of seconds, and θl,0 means an initial phase.
(t)

(t)

In the first flame, θl,0 was set to 0. Fron then on, θl,0 was
given by(t)(t−1)π(Fl −Fl
)

2K

(t−1)

+ θl,0(t−1), where Fl

denotes a(t−1)

frequency of lth overtone in the previous flame and θl,0
denotes an initial phase in the precious flame.

4. Vocal Activities Detection
We remove non-vocal sections using the vocal activities
detection method. The melody waveform obtained withthe accompaniment sound reduction method contains instrumental sounds in non-vocal sections. The existence oflong non-vocal sections negatively influences the executionof the Viterbi alignment between the audio signal and the
lyrics, if interlude sections are not removed.
We propose a vocal activities detection method that cancontrol a balance between hit rate and correct rejection rate.
Generally, there is a trade-off relationship between hit rateand correct rejection rate, and proper balance between them
depends on the application. For example, since our system
positions the vocal activities detection method as a preprocessing of Viterbi alignment, we attach importance to hit
rate instead of correct rejection rate. In other words, wewant to detect all regions that contain vocals. On the otherhands, other applications such as singer identification require maintenance of a high correct rejection rate, and detection of the regions that certainly contain vocals.
In previous vocal activities detection methods [2, 18, 13],no studies have ever tried to control a balance between hit
rate and correct rejection rate.

4.1. Basic Formulation
We introduce a Hidden Markov Model (HMM) that transitions back and forth between vocal state, sV , and nonvocal state, sN , as shown in figure 2. The vocal state meansthat vocals are present and the non-vocal state means that
vocals are absent. Given the feature vectors of input audio
signals, the problem is finding the most likely sequence of
vocal and non-vocal states, Ŝ = {s1 , · · · , st , · · ·}.
Ŝ = argmax{log p(x|st ) + log p(st+1 |st )} , (11)Stwhere p(x|s) represents an output probability of state s, andp(si |sj ) represents a state transition probability from state
sj to state si .

Vocal State sV

Output Probability

Non-Vocal State sN

Output Probabilityp(x | sN )p(x | sV )Figure 2. A Hidden Markov Model (HMM) for
vocal activities detection.

The output log probability of each state is approximatedwith the following equations:1
log p(x|sV ) = log NGMM (x; θV ) − η,21
log p(x|sN ) = log NGMM (x; θN ) + η,
2

(12)
(13)

where NGMM (x; θ) denotes the probability density function of the Gaussian mixture model (GMM) with parameter θ, and η represents a threshold parameter that controls
trade-off between hit rate and correct rejection rate. The parameters of the vocal GMM, θV , and the vocal GMM, θN ,are trained on feature vectors extracted from vocal sections
and nonvocal sections of the training data set, respectively.
We set the number of mixture of the GMM at 64.

• LPC-derived mel cepstral coefficients (LPMCCs):
We use LPC-derived mel cepstral coefficients (LPMCCs) as spectral feature for vocal/non-vocal discrimination. LPMCCs are mel-cepstral coefficients of the
LPC spectrum [1]. We have reported that, in the context of singer identification, LPMCCs represent vocal
characteristics better than mel-frequency cepstral coefficients (MFCCs) [3, 10], which are widely used formusic modeling [4]. We derive LPMCCs by computing the MFCC from the LPC spectrum because of simplicity of implementation.
• ∆F0s:We use the derivative of F0s (∆F0s) [14], which represent the dynamics of the F0’s trajectory, because
singing voice tends to have temporal variation of F0s
because of vibrato and, therefore, ∆F0s are expected
to be good cues for vocal/non-vocal discrimination.
∆F0 is calculated as a regression parameter over fiveframes:2
k · f [t + k]∆f [t] =

k=−2
2


,
k

(16)

2

k=−2

where f [t] represents the frequency in units of cents atthe time t.

5. Viterbi Alignment
4.2. Calculation of Threshold
The balance of vocal activities detection is controlled bychanging η in Equations (12) and (13). However, there isbias in the log likelihoods of GMMs for each song and it
is difficult to decide universal value of η. Therefore, we
divide η into bias correction value, ηdyn. , and application
dependent value, ηﬁxed .
η = ηdyn. + ηﬁxed(14)

While the application dependent value, ηﬁxed , is set byhand, the bias correction value, ηdyn. , is determined by using Otsu’s method for threshold selection [15] as follows.
We first calculate a difference of log likelihood, l(x), of all
the feature vectors in input audio signals.
l(x) = log NGMM (x; θV ) − log NGMM (x; θN ).

(15)

Then we create a histogram of l(x) and select a thresholdto maximize the between-class variance.

In this section, we describe our method of executing
Viterbi alignment between lyrics and segregated signals.
We first create a language model from the given lyrics andthen extract feature vectors from segregated vocal signals.
Finally, we execute the Viterbi alignment between them.
We also describe our method of adapting a phone modelto the specific singer of the input audio signals.

5.1. Lyrics Processing
Given the lyrics corresponding to input audio signals, wecreate a language model for forced alignment. In this language model, we deal with only vowel phonemes because
the unvoiced consonant phonemes do not have a harmonic
structure and cannot be extracted by using the accompaniment sound reduction method. In addition, the voiced consonant phonemes are usually uttered for a short time and it
is difficult to estimate F0s stably. We first convert the lyrics
to a sequence of the phonemes and then create a language
model using the following rules:

4.3. Feature Extraction• Ignore all the consonant phoneme except the syllabic
nasal.

Feature vectors used in this method consist of the following two features.

• Convert each boundary of sentences or phrases into
multiple short pauses.

Proceedings of the Eighth IEEE International Symposium on Multimedia (ISM'06)
0-7695-2746-9/06 $20.00 © 2006

original lyrics (Japanese)

立ち止まる時 またふと振り返る

sequence of the phonemestachidomaru toki mata futo furikaeru
language model

a

i

o

a

u

sp

o

sp

a

a

sp

u

o

sp

u

i

a

e

u

i

Table 1. Evaluation data set from RWC-MDBP-2001.
Song # Singer Name
Gender012
Kazuo Nishi
Male027
Shingo Katsuta
Male
032
Masaki Kuehara
Male
037
Hatae Yoshinori
Male
039
Kousuke Morimoto
Male
007
Tomomi Ogata
Female013
Konbu
Female020
Eri Ichikawa
Female065
Makiko Hattori
Female075
Hiromi Yoshii
FemaleFigure 3. Example of lyrics processing.
feature vectors extracted from segregated vocal signals, andthe adapted phone model for the specific singers. MFCCs[3], ∆ MFCCs, and ∆ power are used as a feature vector
for the Viterbi alignment.
Figure 4. Example of phoneme labels.

6. Experiments• Convert each boundary of words into a single shortpause.
Figure 3 shows an example of conversion from lyrics to thelanguage model.

5.2. Adaptation of a Phone Model
We adapt a phone model to the specific singer of input
audio signals. Our adaptation method consists of the following three steps:
Step 1. We adapt a phone model for clean speech to a clean
singing voice.
Step 2. We adapt the phone model for a clean singing voice
to the singing voice segregated using the accompaniment sound reduction method.
Step 3. We adapt the phone model for segregated speech
to the specific singer of input audio signals using theunsupervised adaptation method.
Steps 1 and 2 are carried out preliminarily, and step 3 iscarried out at runtime.
As an adaptation method, we use MLLR and MAP adaptation. We manually annotated phoneme labels to the adaptation data for supervised adaptation. Figure 4 shows anexample of phoneme labels.

5.3. Alignment
We execute the Viterbi alignment (forced alignment) byusing the language model created from the given lyrics, theProceedings of the Eighth IEEE International Symposium on Multimedia (ISM'06)
0-7695-2746-9/06 $20.00 © 2006

We conducted the experiments to evaluate the performance of our system.

6.1. Experimental Condition
For the evaluation data set, we used the 10 songs listed
in Table 1 taken from “RWC Music Database: Popular Music” (RWC-MDB-P-2001) [6]. These songs are mainly sungin Japanese, but small portion of the vocal part is sung in
English. In this experiments, we approximate the English
phonemes using similar Japanese phonemes. Using these
data, we conducted a 5 fold cross validation for each gender, that is, when we evaluated a song by a particular singer
we adapted a phoneme model using the remaining songs ofthe same gender.
As the training data for the vocal activities detection
method, we used 19 songs of 11 singers listed in Table 2
which are also taken from “RWC Music Database: PopularMusic” (RWC-MDB-P-2001). These 10 singers differ fromthe singers used for evaluation. We applied the accompaniment sound reduction method to these training data. We set
ηﬁxed to 1.5.
Table 3 shows the analysis conditions of the Viterbialignment. As an initial phone model, we used the genderdependent monophone model of ISRC Software [9]. For
conversion from lyrics to a sequence of phonemes, we use
the readings created by ChaSen [11], which is a Japanesemorphological analysis system. For feature extraction, theViterbi alignment, and an adaptation of the phone model,
we use HCopy, HVite,and HAdapt in the Hidden Markov
Toolkit (HTK) [17].

Audio signal

Table 2. Training data for vocal activities detection from RWC-MDB-P-2001.
Singer Name
Gender Piece Number
Hiroshi Sekiya
M048, 049, 051
Katsuyuki Ozawa
M015, 041
Masashi HashimotoM056, 057
Satoshi Kumasaka
M
047
OrikenM
006
Tomoko Nitta
F
026
Kaburagi Akiko
F
055
Yuzu IijimaF
060
Reiko SatoF
063
Tamako Matsuzaka
F070
Donna BurkeF
081, 089, 091, 093, 097

Ground truth annotation (manually labeled)Phrase A

Phrase B

Phrase C

System output

Phrase A

Phrase B

Correct

Incorrect Correct

Accuracy =Phrase C

Incorrect

Correct

Length of “correct” regions
Total length of the song

Figure 5. Evaluation measure.
100
90
80
70
60

Table 3. Analysis conditions of Viterbi alignment.
Sampling
16 kHz, 16 bit
Window function Hamming
Frame length
25 ms
Frame period
10 ms
12th order MFCC
12th order ∆MFCC
Feature vector∆Power

Evaluation was done by using phrase level alignment. In
these experiments, we define phrase as a section that was
delimited by a space or a line feed in the original lyrics. Asan evaluation measure, we calculate proportion of a lengthof the sections that are correctly labeled in phrase level toa total length of a song (Figure 5). The system output ofa song is judged to be satisfactory if its accuracy was over90%.

6.2. Evaluation of Whole System
We conducted experiments using the system in which
all the methods described in this paper were implemented.
Figure 6 shows the results of these experiments.

50
40
30
20
10
0
#012

#027

#032
Male

#037

#039

#007

#013

#020

#065

#075

FemaleFigure 6. Experimental result: evaluation ofwhole system.

did not execute the unsupervised adaptation to input
audio signals.
(iii) Two-step adaptation: First, we adapted a phonemodel for clean speech to clean vocal signals, and thenwe adapted the phone model to segregated vocal signals. We did not execute the unsupervised adaptation
to input audio signals.
(iv) Three-step adaptation (proposed): First, we adapted
a phone model for clean speech to clean vocal signals,
and then we adapted the phone model to segregatedvocal signals. Finally, we adapted the phone model tothe specific singer of input audio signals.

6.3. Evaluation of Adaptation Method

In this experiment, the vocal activities detection method wasenabled. Figure 7 shows the result of these experiments.

The purpose of this experiment was to investigate the effectiveness of the adaptation method. We conducted experiments under the following four conditions:6.4. Evaluation of Vocal Activities Detection(i) No adaptation: We did not execute the phone model
adaptation.
(ii) One-step adaptation: We adapted a phone model forclean speech directly to segregated vocal signals. We

Proceedings of the Eighth IEEE International Symposium on Multimedia (ISM'06)
0-7695-2746-9/06 $20.00 © 2006

The purpose of this experiment was to investigate theeffectiveness of the vocal activities detection method. We
also investigated the performance of the vocal activities detection method. We compared the results of disabling the

100

100

90

90

80

80

70

70

60

60

50

50

40

40

30

30

i. No adaptationii. 1 stepiii. 2 steps
iv. 3 steps

20
10

Hit rate
Correct rejection rate

20
10
0

0
#012 #027 #032 #037 #039 #007 #013 #020 #065 #075
Male

Av.

Female

#012

#027

#032
Male

Figure 7. Experimental result: evaluation ofadaptation.

#037

#039

#007

#013

#020

#065

#075

FemaleFigure 9. Experimental result: hit rate andcorrect rejection rate of vocal activities detection.

100
90
80
70
60
50
40
30
Without detection
With detection

20
10
0

#012 #027 #032 #037 #039 #007 #013 #020 #065 #075
Male

Av.

FemaleFigure 8. Experimental result: evaluation of
vocal activities detection.

vocal activities detection with the results of enabling it. In
this experiment, the three-step adaptation was executed toadapt a phone model to the specific singer of input audio
signals. Figure 8 shows the results of these experiments andFigure 9 shows hit rate and correct rejection rate of the vocal
activities detection method.

6.5. Discussion
As shown in Figure 6, the accuracy was more than 90%
for 8 songs. When we compare the results for the males
and female singers, the accuracy for the female singers islower. This is because it is difficult to capture the characteristics of the voices with high F0 [16]. When we analyzed
errors in each song, typical errors occurred at sections inwhich the lyrics is sung in English. This means that it issometimes difficult to approximate English phonemes using
similar Japanese phonemes. To overcome this problem, weplan to use an English phone model in combination with aJapanese one. Other typical errors were caused by a singing

Proceedings of the Eighth IEEE International Symposium on Multimedia (ISM'06)
0-7695-2746-9/06 $20.00 © 2006

style of humming that was not written in the lyrics.
As shown in Figure 7, our adaptation method was effective for all 10 songs, and as shown in Figure 8, the vocal activities detection method was also effective when applied to the songs with relatively low accuracy. On the other
hand, this method had poor efficacy to the songs #007 and#013, even though the accuracies for these songs are relatively low. This is because correct rejection rates for thesesongs were not so high as shown in Figure 9. In addition,
when this method was applied to songs with higher accuracy, #012 and #037, the accuracy slightly decreased. Thisis because the sections improperly rejected by the vocal activities detection method are always judged incorrect.

7. Conclusions
We have described a system for automatically synchronizing between musical audio signals and corresponding
lyrics. Our system consists of the following three methods:
accompaniment sound reduction, vocal activities detection,
and Viterbi alignment. We also propose a method for adapting a phoneme model to the segregated vocal signals of thespecific singer. Experimental results showed that our system is robust enough to synchronize lyrics with real-worldmusic signals containing sounds of various instruments.
The main contributions of this paper can be summarized
as follows:• We first dealt with the problem of synchronization between music and lyrics forthrightly by segregating vocal signals from sound mixtures and recognizing thephonemes. Due to negative influences caused by accompaniment sounds, no other studies have succeeded
in applying the technique of speech recognition to thistask.
• We proposed an original vocal activities detection
method that can control the trade-off between hit rateand correct rejection rate by changing the parameter.
Although the balance between hit rate and correct rejection rate differs depending on the application, little attention has been given to this viewpoint. We enabled it by dividing the threshold into a bias correction
value and an application dependent value, and obtaining the bias correction value automatically by using
Otsu’s method [15].
• We proposed a method to adapt a phone model for
speech to segregated vocal signals. This method is useful for both alignment between music and lyrics andlyrics recognition of polyphonic audio signals.
In the future, we plan to conduct experiments using
songs sung in languages other than Japanese. We also plan
to incorporate higher-level information such as song structures and achieve more advanced synchronization between
music and lyrics.

8. Acknowledgements
This research was partially supported by the Ministryof Education, Culture, Sports, Science and Technology(MEXT), Grant-in-Aid for Scientific Research (A), No.
15200015, and Information Research Center for Development of Knowledge Society Infrastructure (COE programof MEXT, Japan). We thank Tetsuro Kitahara, Kazuyoshi
Yoshii (Kyoto University) and Tomoyasu Nakano (University of Tsukuba) for their valuable discussions.

References[1] B. S. Atal. Effectiveness of linear prediction characteristicsof the speech wave for automatic speaker identification andverification. the Journal of the Acoustical Society of America, 55(6):1304–1312, 1974.
[2] A. L. Berenzweig and D. P. W. Ellis. Locating singing
voice segments within music signals. In Proceedings IEEE
Workshop on Applications of Signal Processing to Audio andAcoustics (WASPAA), 2001.
[3] S. B. Davis and P. Mermelstein. Comparison of parametric representation for monosyllabic word recognition. IEEE
Transactions on Acoustic, Speech and Signal Processing,28(4):357–366, 1980.
[4] H. Fujihara, T. Kitahara, M. Goto, K. Komatani, T. Ogata,and H. G. Okuno. Singer identification based on accompaniment sound reduction and reliable frame selection. In
Proceedings of the 6th International Conference on Music
Information Retrieval (ISMIR 2005), pages 329–336, 2005.
[5] M. Goto. A real-time music-scene-description system:
predominant-f0 estimation for detecting melody and basslines in real-world audio signals. Speech Communication,
43(4):311–329, 2004.
[6] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka.
RWC Music Database: Popular, classical, and jazz music

Proceedings of the Eighth IEEE International Symposium on Multimedia (ISM'06)
0-7695-2746-9/06 $20.00 © 2006

databases. In Proceedings of the 3rd International Conference on Music Information Retrieval (ISMIR 2002), pages
287–288, October 2002.
[7] T. Hosoya, M. Suzuki, A. Ito, and S. Makino. Lyrics recognition from a singing voice based on finite state automaton
form music information retrieval. In Proceedings of the 6thInternational Conference on Music Information Retrieval(ISMIR 2005), pages 532–535, 2005.
[8] C. kai Wang, R.-Y. Lyu, and Y.-C. Chiang. An automatic
singing transcription system with multilingual singing lyric
recognizer and robust melody tracker. In Proceedings of the
8th European Conference on Speech Communication andTechnology (Eurospeech2003), pages 1197–1200, 2003.
[9] T. Kawahara, A. Lee, K. Takeda, and K. Shikano. Recent
progress of open-source LVCSR engine julius and japanesemodel repository – software of continuous speech recognition consortium –. In Proceedings of The Sixth International Conference on Spoken Language Processing (Interspeech2004 ICSLP), 2004.
[10] B. Logan. Mel frequency cepstral coefficients for musicmodelling. In Proceedings of the International Symposium
on Music Information Retrieval (ISMIR 2000), pages 23–25,
2000.
[11] Y. Matsumoto, A. Kitauchi, T. Yamashita, Y. Hirano,
H. Matsuda, K. Takaoka, and M. Asahara. Japanese morphological analysis system ChaSen. http://chasen.naist.jp/,
2000.
[12] J. A. Moorer. Signal procesing aspects of computer music:
A survey. Proceedings of the IEEE, 65(8):1108–1137, 1977.
[13] T. L. Nwe and Y. Wang. Automatic detection of vocal segments in popular songs. In Proceedings of the 5th International Conference on Music Information Retrieval (ISMIR
2004), pages 138–145, 2004.
[14] Y. Ohishi, M. Goto, K. Itou, and K. Takeda. Discrimination between singing and speaking voices. In Proceedingsof 9th European Conference on Speech Communication andTechnology (Eurospeech 2005), pages 1141–1144, 2005.
[15] N. Otsu. A threshold selection method from gray-level histograms. IEEE Transaction on System, Man, and Cybernetics, SMC-9(1):62–66, 1979.
[16] A. Sasou, M. Goto, S. Hayamizu, and K. Tanaka. An autoregressive, non-stationary excited signal parameter estimation method and an evaluation of a singing-voice recognition. In Proceedings of the 2005 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP
2005), pages I–237–240, 2005.
[17] The
HiddenMarkov
Model
Toolkit(HTK),
http://htk.eng.cam.ac.uk/.
[18] W.-H. Tsai and H.-M. Wang. Automatic detection and tracking of target singer in multi-singer music recordings. In
Proceedings of the 2004 IEEE International Conference on
Acoustics, Speech, and Signal Processing (ICASSP 2004),
pages 221–224, 2004.
[19] Y. Wang, M.-Y. Kan, T. L. Nwe, A. Shenoy, and J. Yin. Lyrically: Automatic synchronization of acoustic musical signals and textual lyrics. In Proceedings of the 12th ACM
International Conference on Multimedia, pages 212–219,
2004.