1252

IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 5, NO. 6, OCTOBER 2011

LyricSynchronizer: Automatic Synchronization
System Between Musical Audio Signals and Lyrics
Hiromasa Fujihara, Masataka Goto, Jun Ogata, and Hiroshi G. Okuno

Abstract—This paper describes a system that can automaticallysynchronize polyphonic musical audio signals with their corresponding lyrics. Although methods for synchronizing monophonicspeech signals and corresponding text transcriptions by using
Viterbi alignment techniques have been proposed, these methods
cannot be applied to vocals in CD recordings because vocals areoften overlapped by accompaniment sounds. In addition to aconventional method for reducing the influence of the accompaniment sounds, we therefore developed four methods to overcome
this problem: a method for detecting vocal sections, a method forconstructing robust phoneme networks, a method for detecting
fricative sounds, and a method for adapting a speech-recognizer
phone model to segregated vocal signals. We then report experimental results for each of these methods and also describe ourmusic playback interface that utilizes our system for synchronizing music and lyrics.
Index Terms—Alignment, lyrics, singing voice, Viterbi algorithm, vocal.

I. INTRODUCTIONS

INCE the lyrics of a song represent its theme and story, they
are essential to creating an impression of the song. This
is why music videos often help the audience enjoy the musicby displaying synchronized lyrics as a caption. When a song isheard, for example, some people listen to the vocal melody andfollow the lyrics.
In this paper, we describe a system that synchronizes the polyphonic audio signals and the lyrics of songs automatically by
estimating the temporal relationship (alignment) between theaudio signals and the corresponding lyrics. This approach is different from direct lyrics recognition and takes advantage of thevast selections of lyrics available on the web. Our system has anumber of applications, such as automatic generation of music
video captions and a music playback interface that can directly
access to specific words or passages of interest.
Wang et al. developed a system called LyricAlly [1] forsynchronizing lyrics with music recordings without extracting
singing voices from polyphonic sound mixtures. It uses theManuscript received September 30, 2010; revised March 01, 2011; accepted
May 18, 2011. Date of publication June 16, 2011; date of current version
September 16, 2011. This work was supported in part by CrestMuse, CREST,
JST. The associate editor coordinating the review of this manuscript andapproving it for publication was Prof. Daniel Ellis.
H. Fujihara, M. Goto, and J. Ogata are with the National Institute of Advanced
Industrial Science and Technology (AIST), Tsukuba 305-8568, Japan (e-mail:h.fujihara@aist.go.jp; m.goto@aist.go.jp; jun.ogata@aist.go.jp).
H. G. Okuno is with Kyoto University, Kyoto 606-8501, Japan (e-mail:okuno@i.kyoto-u.ac.jp).
Digital Object Identifier 10.1109/JSTSP.2011.2159577

duration of each phoneme as a cue for synchronization but it is
not always effective because phoneme duration varies and canbe altered by musical factors such as the location in a melody.
Wong et al. [2] developed an automatic synchronization system
for Cantonese popular music. It uses the tonal characteristics ofCantonese language and compares the tone of each word in the
lyrics with the fundamental frequency (F0) of the singing voice,but because most languages do not have the tonal characteristics of Cantonese, this system cannot be generalized to mostother languages. Loscos et al. [3] used a speech recognizer
for aligning a singing voice and Wang et al. [4] used a speech
recognizer for recognizing a singing voice, but they assumed
pure monophonic singing without accompaniment. Gruhne etal. [5] worked on phoneme recognition in polyphonic music.
Assuming that boundaries between phonemes were given, they
compared several classification techniques. Their experiments
were preliminary, and there were difficulties in actually recognizing the lyrics.
Since current speech recognition techniques are incapableof automatically synchronizing lyrics with music that includes
accompaniment, we used an accompaniment sound reduction
method [6] as well as the following four methods: a methodfor detecting vocal sections, a method for detecting fricative
sounds, a method for constructing a phoneme network that isrobust to utterances not in the lyrics, and a method for adapting
a phone model for speech to segregated vocal signals.

II. SYSTEM FOR AUTOMATICALLY SYNCHRONIZINGMUSIC AND LYRICS
Given musical audio signals and the corresponding lyrics, oursystem calculates the start and end times for each phoneme of
the lyrics. The target data are real-world musical audio signals
such as popular music CD recordings that contain a singer’s
vocal track and accompaniment sounds. We make no assumptions about the number and kind of sound sources in the accompaniment sounds. We assume that the main vocal part is sung bya single singer (except for choruses).
Because the ordinary Viterbi alignment (forced alignment)
method used in automatic speech recognition is negatively influenced by accompaniment sounds performed together with a
vocal and also by interlude sections in which the vocal is not
performed, we first obtain the waveform of the melody by extracting and resynthesizing the harmonic structure of the melody
using the accompaniment sound reduction method proposed in[6]. We then detect the vocal region in the separated melody’s
audio signal, using a vocal activity detection method based ona hidden Markov model (HMM). We also detect the fricative

1932-4553/$26.00 © 2011 IEEE

FUJIHARA et al.: LYRICSYNCHRONIZER: AUTOMATIC SYNCHRONIZATION SYSTEM

1253

Fig. 1. Overview of accompaniment sound reduction.

sound by using a fricative sound detection method and incorporate this information into the next alignment stage. Finally, wealign the lyrics and the separated vocal audio signals by usinga Viterbi alignment method. The language model used in thisalignment stage incorporates a filler model so that the system
becomes robust to inter-phrase vowel utterances not written in
the lyrics. We also propose a method for adapting a phone modelto the separated vocal signals of the specific singer.
A. Accompaniment Sound Reduction
To extract a feature that represents the phonetic informationof a singing voice from polyphonic audio signals, we need toreduce the accompaniment sound, as shown in Fig. 1. We do thisby using a melody resynthesis technique based on a harmonic
structure [6] consisting of the following three parts:
1) estimate the fundamental frequency (F0) of the melody byusing Goto’s PreFEst [7];2) extract the harmonic structure corresponding to the
melody;3) resynthesize the audio signal (waveform) corresponding to
the melody by using a sinusoidal synthesis.
We thus obtain a waveform corresponding only to the melody.
Fig. 2 shows spectrograms of polyphonic musical audio signals, that of the audio signals segregated by the accompanimentsound reduction method, and that of the original (ground-truth)
vocal-only signals. It can be seen that the harmonic structure ofa singing voice is enhanced by using the accompaniment soundreduction method.
Note that the melody obtained this way contains instrumental(i.e., nonvocal) sounds in interlude sections as well as voices invocal sections, because the melody is defined as merely the most
predominant note in each frame [7]. Since long nonvocal sections negatively influence the execution of the Viterbi alignment
between the audio signal and the lyrics, we need to remove theinterlude sections. Vocal sections are therefore detected by using
the method described in Section II-B. Furthermore, since thismethod is based on the harmonic structure of the singing voice,unvoiced consonants, which do not have harmonic structures,
cannot be separated properly. We try to partially overcome thisFig. 2. Example of accompaniment sound reduction taken from [6]. (a) A spectrogram of polyphonic signals. (b) A spectrogram of segregated signals. (c) A
spectrogram of vocal-only signals.

issue by using the fricative sound detection method described in
Section II-C.
Since the accompaniment sound reduction method is executed as a preprocessing of feature extraction for Viterbi alignment, it is easy to replace this with other singing voice separation or an F0 estimation method [8]. In this paper, we adopt
the PreFEst-based accompaniment sound reduction method because it was reported that PreFEst achieved higher performancein F0 estimation experiments of polyphonic singing voices [9].
1) F0 Estimation: We used Goto’s PreFEst [7] to estimate
the F0 of the melody line. PreFEst can estimate the most predominant F0 in frequency-range-limited sound mixtures. Since
the melody line tends to have the most predominant harmonic
structure in middle- and high-frequency regions, we can estimate the F0 of the melody line by applying PreFEst with adequate frequency-range limitations.
2) Harmonic Structure Extraction: By using the estimatedF0, we then extract the amplitude of the fundamental frequency

1254IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 5, NO. 6, OCTOBER 2011

The output log probability of each state is approximated withthe following equations:(4)(5)
Fig. 3. Hidden Markov model (HMM) for vocal activity detection.

component and harmonic components. For each component, weallow cent1 error and extract the local maximum amplitude in
the allowed area. The frequency
and amplitudeof theth overtone
at timecan be represented as

(1)
(2)
where
denotes the complex spectrum, anddenotes
F0 estimated by the PreFEst. In our experiments, we set to 20.
3) Resynthesis: Finally, we use a sinusoidal model to resynthesize the audio signal of the melody by using the extracted
and. Changes in phase are apharmonic structure,proximated using a quadratic function so that the frequency canchange linearly. Changes in amplitude are also approximatedusing a linear function.
B. Vocal Activity Detection
We propose a vocal activity detection method that can control
the balance between the hit and correct rejection rates. There is
generally a tradeoff relationship between the hit and correct rejection rates, and a proper balance between them depends on theapplication. For example, since our system positions the vocal
activity detection method before the Viterbi alignment, the hit
rate is more important than the probability of correct rejection
because we want to detect all the regions that contain vocals. No
previous studies on vocal activity detection [10]–[12] ever tried
to control the balance between the probabilities.
1) Basic Formulation: We introduce a hidden Markov model(HMM) that transitions back and forth between vocal state, ,, as shown in Fig. 3. Vocal state means
and non-vocal state,that vocals are present and non-vocal state means that vocals
are absent. Given the feature vectors of input audio signals, theproblem is finding the most likely sequence of vocal and non:vocal states,
(3)

where

represents an output probability of state , andrepresents a state transition probability for the tran.
sition from state to state
1The cent is a logarithmic scale used for musical intervals in which the octave
is divided into 1200 cents.

denotes the probability density function of
where
the Gaussian mixture model (GMM) with parameter , andrepresents a threshold parameter that controls tradoff between
the hit and correct rejection rates. The parameters of the vocal
GMM, , and the nonvocal GMM, , are trained on feature
vectors extracted from vocal sections and nonvocal sections of
the training data set, respectively. We set the number of GMM
mixtures to 64.
2) Calculation of Threshold: The balance of vocal activity
detection is controlled by changing in (4) and (5), but there isbias in the log likelihoods of the GMMs for each song, and it isdifficult to decide the universal value of . We therefore divide
into a bias correction value,, and an application-dependent
value,:(6), is obtained from input audio
The bias correction value,
signals by using Otsu’s method for threshold selection [13]based on discriminant analysis, and the application-dependent, is set by hand.
value,
We first calculate the difference of log likelihood,, for allthe feature vectors in input audio signals:(7)
We then calculate the bias correction value,, by using
Otsu’s method. The Otsu’s method assumes that a set ofcontains two classes of values and calculates the optimum
threshold that maximizes their inter-class variance. When ais denoted as, the inter-class variancehistogram of
can be written as
(8)
(9)
(10)
(11)
, can take only a finite number
In practice, the threshold,of values since the histogram,, is a discrete function. Thus,
for all possible
and
it is possible to calculate
obtain the optimum value.
3) Novel Feature Vectors for Vocal Activity Detection:
The vocal activity detection after the accompaniment soundreduction can be interpreted as a problem of judging whether
the sound source of the given harmonic structure is vocal or
nonvocal. In our previous system [6], we estimated the spectralenvelope of the harmonic structure and evaluate the distance
between it and the spectral envelopes in the training database.

FUJIHARA et al.: LYRICSYNCHRONIZER: AUTOMATIC SYNCHRONIZATION SYSTEM

1255

However, spectral envelopes estimated from high-pitchedsounds by using cepstrum or linear prediction coding (LPC)
analysis are strongly affected by spectral valleys between
adjacent harmonic components. Thus, there are some songs(especially those sung by female singers) for which the vocal
activity detection method did not work well.
This problem boils down to the fact that a spectral envelope estimated from a harmonic structure is not reliable except for the points (peaks) around each harmonic component.
This is because a harmonic structure could correspond to different spectral envelopes: the mapping from a harmonic structure to its original spectral envelopes is a one-to-many association. When we consider this issue using sampling theory, theharmonic components are points sampled from their original
spectral envelope at the interval of F0 along the frequency axis.
The perfect reconstruction of the spectral envelope from the harmonic components is therefore difficult in general. Because conventional methods, such as Mel-frequency cepstral coefficient(MFCC) and LPC, estimate only one possible spectral envelope,the distance between two sets of the harmonic structure from thesame spectral envelope is sometimes inaccurate. Though several
studies have been proposed that have tried to overcome such instability of cepstrum [14], [15] by interpolating harmonic peaks
or introducing new distance measures, such studies still havebeen trying to estimate a spectral envelope from an unreliable
portion of the spectrum.
To overcome this problem, the distance must be calculatedusing only the reliable (sampled) points at the harmonic components. We focus on the fact that we can directly compare thepower of harmonic components between two sets of the harmonic structure if their F0s are approximately the same. Our approach is to use the power of harmonic components directly as
feature vectors and compare the given harmonic structure with
only those in the database that have similar F0 values. This approach is robust against high-pitched sounds, because the spectral envelope does not need to be estimated. The powers of firstto 20th overtones from the polyphonic audio signals are extracted and used as a feature vector.
To ensure that comparisons are made only with feature vectors that have similar F0s, we also use the F0 value as a feature in
addition to the power of harmonic components. By using GMMs
to model the feature vectors, we can be sure that each Gaussian
can cover feature vectors that have similar F0s. When we calculate the likelihood of a GMM, the weights of the Gaussians
that have large F0 values are minuscule. Thus, we can calculate the distance only with harmonic structures that have similar
F0 values. There have been studies that used similar features in
the field of sound source recognition [16]. These studies concern instrumental sounds, and it is not derived from an aspect ofspectral envelope estimation.
The absolute value of the power of the harmonic structureis biased depending on the volume of each song. We therefore
normalize the power of all harmonic components for each song.
The normalized power of the th harmonic component at time
, , is given by

where represents the original power, is the total number offrames, and is the number of harmonic components considered. In this equation, an average power of every frequency binof all the frames is subtracted from the original power in a log
domain.

The forced alignment algorithm used in automatic speech
recognition (ASR) synchronizes speech signals and texts bymaking phoneme networks that consist of all the vowels andconsonants. However, since the accompaniment sound reduction, which is based on the harmonic structure of the melody,cannot segregate unvoiced consonants that do not have harmonic
structure, it is difficult for the general forced alignment algorithm to align unvoiced consonants correctly unless we introduce a method for detecting unvoiced consonants from the original audio signals. We therefore developed a signal processing
technique for detecting candidate unvoiced fricative sounds (atype of unvoiced consonant) in the input audio signals. Here,
we focus on the unvoiced fricative sounds because their durations are generally longer than those of the other unvoiced consonants and because they expose salient frequency components
in the spectrum.
1) Nonexistence Region Detection: It is difficult to accurately
detect the existence of each fricative sound because the acoustic
characteristics of some instruments (cymbals and snare drums,
for example) sometimes resemble those of fricative sounds. If
we take an approach such that we align /SH/ phoneme to frames
if and only if they were detected as fricative regions, detection
errors (no matter if they are false positive or false negative) candegrade the accuracy significantly in the later forced alignment
step. We therefore take the opposite approach and try to detect
regions in which there are no fricative sounds, i.e., nonexistence
regions. Then, in the forced alignment, fricative consonants areprohibited from appearing in the nonexistence regions. However, if the frames including the /SH/ sound are erroneously
judged as the nonexistence region, this kind of error affects theperformance even in this approach; we can ameliorate this influence by setting a strict threshold and having a fricative detector
to detect fewer regions as nonexistence regions.
2) Fricative Sound Detection: Fig. 4 shows an example
spectrogram depicting non-periodic source components suchas snare drum, fricative, and high-hat cymbal sounds in popular
music. The characteristics of these non-periodic source components are depicted as vertical lines or clouds along the frequency
axis in the spectrogram, whereas periodic source components
tend to have horizontal lines. In the frequency spectrum at a
certain time, these vertical and horizontal lines, respectively,
correspond to flat and peaked (pointed) components.
To detect flat components from non-periodic sources, weneed to ignore peak components in the spectrum. We therefore use the bottom envelope estimation method proposed by
Kameoka et al. [17]. As shown in Fig. 5, the bottom envelope
is defined as the envelope curve that passes through spectral
valleys. The function class of the bottom envelope is defined as

(12)

(13)

C. Use of Unvoiced Consonants Based on Fricative Detection

1256IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 5, NO. 6, OCTOBER 2011

to the power of most other bands. Since the sampling rate in ourcurrent implementation is 16 kHz, we deal with only the unvoiced fricative phoneme /SH/ because we found from our observations that the other unvoiced fricative phonemes tended tohave much power in the frequency region above 8 kHz, which is
the Nyquist frequency of 16-kHz sampling. Since the phoneme/SH/ has strong power in the frequency region from 6 to 8 kHz,
we define the existence degree of the phoneme /SH/ as follows:
Fig. 4. Example spectrogram depicting snare drum, fricative, and high-hat
cymbal sounds.

(17)
is below a threshold (0.4) are identified
Regions in which
as nonexistence regions, where phoneme /SH/ does not exist.
The threshold 0.4 was determined experimentally. Note that to
avoid any effect from bass drums, we do not use frequency com.
ponents below 1 kHz in the calculation of
D. Viterbi Alignment

Fig. 5. Bottom envelope g (f; a ) in a spectrum S (f ).

wheredenotes the frequency in Hz,
is theGaussian function, andrepresents the weightsof each Gaussian. This function class approximates arbitral
spectral envelopes by using the weighted sum of Gaussianfunctions of which the means and variances are fixed. The
means of the Gaussians are set so that they equally align to thefrequency axis, and their variances are set so that the shape of
this function class becomes smooth.
The problem here is to estimate , which determines the envelope curve. We therefore estimate the that minimizes theobjective function
(14)
where
represents the spectrum at each frame. This objecandin the Itakura–Saito
tive is derived by reversing
distance. Unlike the Itakura–Saito distance that penalizes positive errors much more than negative ones and is used to estimate
the top envelope of a spectrum, this objective function penalizes negative errors much more than positive ones to estimate
the bottom envelope. From this objective function, we can derive the following iterative equations to obtain :
(15)
(16)
where is the value estimated in the previous iteration. In thisis obtained asway, the bottom envelope of the spectrum.
Among the various unvoiced consonants, unvoiced fricative
sounds tend to have frequency components concentrated in aparticular frequency band of the spectrum. We therefore detect
the fricative sounds by using the ratio of the power of that bandIn this section, we describe our method of executing Viterbialignment between lyrics and separated signals. We first createa language model from the given lyrics and then extract feature
vectors from separated vocal signals. Finally, we execute theViterbi alignment between them. We also describe our methodof adapting a phone model to the specific singer of the input
audio signals.
1) Lyrics Processing Using the Filler Model: Given the lyrics
corresponding to input audio signals, we create a phoneme network for forced alignment. This network basically does not havea branch. By using this network as a language model of a speech
recognition system and calculating the most likely path of a sequence of the feature vectors extracted from the audio signals
based on the Viterbi search algorighm, the start and end timesof each node of network, which correspond to a phoneme inthe lyrics, can be estimated. Note that nodes in the network
are replaced by the HMMs of corresponding phonemes in thephoneme model. Thus, we can align the lyrics with the audio
signals. In our system, since we only have the phoneme modelfor the Japanese language, English phonemes are substituted
with the most similar Japanese phoneme.
We first convert the lyrics to a sequence of phonemes and thencreate a phoneme network by using the following rules:• convert the boundary of a sentence or phrase into multiple
appearances of short pauses (SPs);• convert the boundary of a word into one appearance of an
SP.
Fig. 6 shows an example of conversion from lyrics to the language model.
Some singers often sing words and phrases not in the actual
lyrics, such as “Yeah” and “La La La,” during interlude sections
and rests between phrases in the lyrics. We found in our preliminary experiments that such inter-phrase vowel utterances reduced the accuracy of the system because the system inevitablyaligned other parts of the lyrics to those utterances. This shortcoming can be eliminated by introducing the filler model [18],[19], which is used in keyword-spotting research.
Fig. 7 is a filler model that we used in this paper. The
five nodes in the figure (a, i, u, e, and o) are Japanese vowelFUJIHARA et al.: LYRICSYNCHRONIZER: AUTOMATIC SYNCHRONIZATION SYSTEM

1257

Fig. 8. Example of phoneme labeling.
TABLE IEVALUATION DATA SET
Fig. 6. Example of conversion from original lyrics to a phoneme network. Original lyrics are converted to a sequence of the phonemes first, then a phoneme
network is constructed from the sequence. Note that sp represents a short pause.
This lyrics was taken from the song No. 100 in RWC-MDB-P-2001.

Fig. 7. Filler model inserted at each phrase boundary in the lyrics.

phonemes. This model is inserted in the middle of two consecutive phrases in the phoneme network. For example, in Fig. 6,
the multiple appearance of sp between the 12th phoneme and13th phoneme (both are /NN/) will be replaced by the filler
model in Fig. 7. If there are utterances that are not written in
the lyrics at that part, vowel nodes of the filler model (a, i, u, e,and o) appear here and reduce the influence of such utterances.
On the other hand, if there is not such utterance vowel nodes ofthe filler model (a, i, u, e, and o) are ignored and the most likely
path connects the two phrases via the /SP/ model.
In our preliminary experiments without using this filler
model, we expected the SPs to represent short nonvocal sections. However, if the singer sang words not in the lyrics innonvocal sections, the SPs, which were originally trained using
nonvocal sections, were not able to represent them. Thus, lyrics
from other parts were incorrectly allocated to these nonvocal
sections. The vowels from the filler model can cover theseinter-phrase utterances.
2) Adaptation of a Phone Model: We adapt a phone modelto the specific singer of input audio signals. As an initial phonemodel, we use a monophone model for speech, since creating aphone model for a singing voice from scratch requires a large annotated training database and this type of a database of singing
voices has not yet been developed. Our adaptation method consists of the following three steps:
Step 1) adapt a phone model for clean speech to a clean
singing voice;
Step 2) adapt the phone model for a clean singing voice to thesinging voice separated using the accompanimentsound reduction method;
Step 3) adapt the phone model for separated speech to thespecific singer of input audio signals by using theunsupervised adaptation method.
Steps 1 and 2 are carried out preliminarily, and step 3 is carriedout at runtime.
As an adaptation method, we use MLLR [20] and MAP [21],which are commonly used in speech recognition research. We
manually annotated phoneme labels to the adaptation data forsupervised adaptation. Fig. 8 shows an example of phoneme
labeling.
3) Alignment: Using the language model created from thegiven lyrics, the feature vectors extracted from separated vocal
signals, and the adapted phone model for the specific singers, we
execute the Viterbi alignment (forced alignment). In this alignment process, we do not allow any phoneme except /SP/ to appear in the nonvocal region and do not allow the phoneme /SH/
to appear in the region of fricative sound nonexistence. MFCCs[22] and derivatives of the MFCCs and power are used as feature vectors for the Viterbi alignment.
III. EXPERIMENTS
A. Experimental Condition
The performance of our system was evaluated experimentally. As an evaluation data set ten Japanese songs by ten singers(five male, five female) were used as listed in Table I. The songs
were taken from the “RWC Music Database: Popular Music”(RWC-MDB-P-2001) [23]. They were largely in Japanese, butsome phrases in their lyrics were in English. In these experiments, English phonemes were approximated by using similar
Japanese phonemes. We conducted a five-fold cross-validation.
We used as the training data for the vocal activity detection
method 19 songs also taken from the RWC-MDB-P-2001, sungby the 11 singers listed in Table II. These singers differed fromthe singers used for evaluation. We applied the accompanimenttosound reduction method to the training data and we set
1.5.
Table III shows the analysis conditions for the Viterbi alignment. As an initial phone model, we used the gender-independent monophone model developed by the IPA Japanese Dictation Free Software Project and Continuous Speech Recognition
Consortium (CSRC) [24]. To convert the lyrics to a sequence ofphonemes, we used Mecab [25], which is a Japanese morphological analysis system.
The evaluation was performed by using phrase level alignment. In these experiments, we defined a phrase as a sectionthat was delimited in the original lyrics by a space or a line feed.

1258

IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 5, NO. 6, OCTOBER 2011

TABLE IITRAINING DATA FOR VOCAL ACTIVITY DETECTIONTABLE III
CONDITIONS FOR ANALYSIS OF VITERBI ALIGNMENT

Fig. 9. Evaluation measure in the experiments on the synchronization of music
and lyrics.

TABLE IV
EXPERIMENTAL RESULTS (%): EVALUATION OF ACCOMPANIMENTSOUND REDUCTION METHODSong number in the RWC-MDB-P-2001.

This is because it is hard to capture the characteristics of voiceswith a high F0 [26]. Analyzing the errors in each song, wefound that errors typically occurred at the sections in which the
lyrics were sung in English. Using similar Japanese phonemes
to approximate English phonemes thus seemed to be difficult.
To overcome this problem, we will try to use an English phonemodel in combination with a Japanese one.
In addition to the above evaluation, we also conducted another evaluation based on a morpheme label ground truth to see
how well the system performed at the morpheme level. We prepared morpheme label annotations for songs No. 12 and No. 20,
calculated the accuracies using the results of the above experiment. An evaluation measure is the same as that explained inFig. 9, except that morphemes were used instead of phrases. The
accuracy for No. 12 was 72.4% and that for No. 20 was 65.3%.
From these results, we can see that our system still achieved
performance above 65%, though there was a certain number ofinevitable decreases.
C. Evaluation of Accompaniment Sound Reduction Method

Fig. 10. Experimental results: evaluation of the whole system.

The evaluation measure we used was the ratio of the total lengthof the sections labeled correctly at the phrase level to the total
length of a song (Fig. 9).
B. Evaluation of the Whole System
We conducted experiments using a system in which all of
the methods described in this paper were implemented. Fig. 10shows the results of these experiments.
When we compare the results in Fig. 10 between male and female singers, we see that the accuracy for the females is lower.

In our experimental evaluation of the accompaniment soundreduction, we disabled the vocal activity detection, fricative detection, and filler model and we enabled the three-step adaptation. We compared the following two conditions: 1) MFCC
extracted from segregated singing voice using accompanimentsound reduction method and 2) MFCC extracted directly frompolyphonic music without using accompaniment sound reduction method. Note that condition 1) in this experiment is thesame as condition 4) in the experiment in Section III-F. We cansee in Table IV that the accompaniment sound reduction improved the accuracy by 4.8 percentage points.
D. Evaluation of Vocal Activity Detection, Fricative Detection,
and Filler Model
The purpose of this experiment was to investigate the separate
effectiveness of the fricative detection, filler model, and vocal
activity detection. We tested our method under five conditions.
1) Baseline: Only the three-step adaptation was enabled.
2) VAD: Only vocal activity detection and the three-step
adaptation were enabled (Section II-B3).
3) Fricative detection: Only fricative sound detection and thethree-step adaptation were enabled (Section II-C).

FUJIHARA et al.: LYRICSYNCHRONIZER: AUTOMATIC SYNCHRONIZATION SYSTEMTABLE V
EXPERIMENTAL RESULTS (%): EVALUATION OF FRICATIVE DETECTION, FILLER
MODEL AND VOCAL ACTIVITY DETECTION

1259

TABLE VI
EXPERIMENTAL RESULTS (%): EVALUATION OF ACCOMPANIMENTSOUND REDUCTION METHODSong number in RWC-MDB-P-2001.
Song number in RWC-MDB-P-2001.

4) Filler model: Only filler model and the three-step adaptation were enabled (Section II-D1).
5) Proposed: The fricative-sound detection, the filler model,
the vocal-activity detection, and the three-step adaptation
were enabled.
We see in Table V that vocal-activity detection, the fricative
detection, and the filler model increased the average accuracy
by 13.0, 0.7, and 1.0 percentage points, respectively, and thatthe highest accuracy, 85.2%, was obtained when all three wereused. The vocal activity detection was the most effective of thethree techniques. Inspection of the system outputs obtained withthe filler model showed that the filler model was effective not
only for utterances not in the actual lyrics, but also for nonvocal regions that could not be removed by vocal activity detection. Since our evaluation measure was phrase-based, the effectiveness of fricative detection could not be fully evaluated.
Inspection of the phoneme-level alignment results showed thatphoneme gaps in the middle of phrases were shorter than they
were without fricative detection. We plan to develop a measurefor evaluating phoneme-level alignment.
E. Evaluation of Feature Vector for Vocal Activity Detection
In our experimental evaluation of the feature vectors for vocal
activity detection, we disabled the fricative detection and filler
model and enabled the three-step adaptation. We compared theeffectiveness of 1) the novel feature vector based on the powerof harmonic structure described in Section II-B3 with that of the
LPMCC-based feature vector proposed in [6]. We also compare
receiver operating characteristic (ROC) curves of these two conditions. Note that condition 2) in this experiment is same as thecondition 3) in the experiment in Section III-D.
We can see in Table VI that the accuracy obtained with thenovel feature vector proposed in this paper was 4.0 percentage
points better than that obtained with the LPMCC-based feature
vector.
Fig. 11 shows the ROC curves of our vocal activity detection system. By changing the application-dependent threshold,, to the various values, various pairs of the hit rates andfalse alarm rates are plotted. The vertical and horizontal axes
represents the hit rate and false alarm rate, respectively. Notethat these rates are calculated by using all the ten songs. Alsofrom this figure, we can see that our new feature vector improves
the accuracy of vocal activity detection.

Fig. 11. Comparison of ROC curves.

F. Evaluation of Adaptation Method
In our experimental evaluation of the effectiveness of theadaptation method, we disabled the vocal activity detection,
fricative detection, and filler model and we conducted experiments under the following four conditions.
1) No adaptation: We did not execute phone model adaptation.
2) One-step adaptation: We adapted a phone model for clean
speech directly to separated vocal signals. We did not execute an unsupervised adaptation to input audio signals.
3) Two-step adaptation: First, we adapted a phone model forclean speech to clean vocal signals, and then we adapted
the phone model to separated vocal signals. We did not
execute an unsupervised adaptation to input audio signals.
4) Three-step adaptation (proposed): First, we adapted aphone model for clean speech to clean vocal signals, thenwe adapted the phone model to separated vocal signals, andfinally we adapted the phone model to the specific singer
of input audio signals.
We can see in Table VII that our adaptation method was effective
for all ten songs.
IV. LYRICSYNCHRONIZER: MUSIC PLAYBACK INTERFACEWITH SYNCHRONIZED-LYRICS-DISPLAY
Using our method for synchronizing music and lyrics, we developed a music playback interface called LyricSynchronizer.
This interface can display the lyrics of the song synchronized

1260IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 5, NO. 6, OCTOBER 2011

TABLE VII
EXPERIMENTAL RESULTS (%): EVALUATION OF ADAPTATION METHOD

A song number of the RWC-MDB-P-2001.

Fig. 12. Screenshot of our music playback interface.

with the music playback. It also has a function that enables users
to jump to a phrase of interest by clicking on the lyrics. Fig. 12shows a screen shot of the interface.
The diffusion of the personal computer and the portable music
player has increased our opportunities to listen to songs while
using devices that have a display. It might be natural to consider
using that display to enrich users’ experience in music appreciation. Most devices with a display show bibliographic information such as the name of the song and the performing artist, andmusic players on personal computers sometimes have visualizer
functions that display animations created from the spectrum of
the music.
Focusing on lyrics as information that should be displayed,
we developed a music playback interface that has the following
two functions: displaying-synchronized-lyrics function, andjump-by-clicking-the-lyrics function. The former function
displays the current position of the lyrics as shown in Fig. 12.
Although this function resembles the lyrics display for karaoke,
manually labeled temporal information is required in the lyrics
display in karaoke. By the latter function, users can change
the current playback position by clicking a phrase in the lyrics
that are displayed. This function is useful when users want tolisten only to sections of interest to them. This function canbe considered as an implementation of active music listening
interfaces [27].

V. CONCLUSION
We have described a system for automatically synchronizing
musical audio signals and their corresponding lyrics. For accurate synchronization we segregate the singing voice and theaccompaniment sound. We also developed a robust phoneme
network using a filler model and developed methods for detecting vocal activity and fricative sound detection for adapting
a phoneme model to the separated vocal signals of a specificsinger. Experimental results showed that our system can accurately synchronize musical audio signals and their lyrics.
In our vocal activity detection method, the tradeoff between
hit rate and correct rejection rate can be adjusted by changing
a parameter. Although the balance between hit rate and correct
rejection rate differs depending on the application, little attention has been given to this tradeoff in past research. Our vocal
activity detection method makes it possible to adjust the tradeoff
based on Otsu’s method [13]. The novel feature vectors basedon the F0 and the power of harmonic components were robust
to high-pitched sounds because a spectral envelope did not need
to be estimated. The underlying idea of the fricative detection(i.e., the detection of nonexistence regions) is a novel one. Experimental evaluation showed that synchronization performance
was improved by integrating this information, even if it was difficult to accurately detect each fricative sound. Although thefiller model is a simple idea, it worked very efficiently because
it did not allow a phoneme in the lyrics to be skipped and it
appeared only when it was needed. We proposed a method foradapting a phone model for speech to separated vocal signals.
This method was useful for music and lyric alignment as well
as for recognizing lyrics in polyphonic music.
We plan to incorporate higher-level information such as song
structures and thereby achieve more advanced synchronization
between music and lyrics. We also plan to expand our music
playback interface, LyricSynchronizer, by incorporating otherelement of music besides lyrics and develop more advanced active music listening interfaces that can enhance music listening
experiences of users.
REFERENCES[1] Y. Wang, M.-Y. Kan, T. L. Nwe, A. Shenoy, and J. Yin, “Lyrically: Automatic synchronization of acoustic musical signals and textual lyrics,”
in Proc. 12th ACM Int. Conf. Multimedia, 2004, pp. 212–219.
[2] C. H. Wong, W. M. Szeto, and K. H. Wong, “Automatic lyrics alignment for Cantonese popular music,” Multimedia Syst., vol. 4–5, no. 12,
pp. 307–323, 2007.
[3] A. Loscos, P. Cano, and J. Bonada, “Low-delay singing voice alignment to text,” in Proc. Int. Comput. Music Conf. (ICMC99), 1999.
[4] C.-K. Wang, R.-Y. Lyu, and Y.-C. Chiang, “An automatic singing transcription system with multilingual singing lyric recognizer and robust
melody tracker,” in Proc. 8th Euro. Conf. Speech Commun. Technol.
(Eurospeech’03), 2003, pp. 1197–1200.
[5] M. Gruhne, K. Schmidt, and C. Dittmar, “Phoneme recognition in popular music,” in Proc. 8th Int. Conf. Music Inf. Retrieval (ISMIR’07),
2007, pp. 369–370.
[6] H. Fujihara, M. Goto, T. Kitahara, and H. G. Okuno, “A modeling of
singing voice robust to accompaniment sounds and its application tosinger identification and vocal-timbre-similarity based music information retrieval,” IEEE Trans. Audio, Speech, Lang. Process., vol. 18, no.
3, pp. 638–648, Mar. 2010.
[7] M. Goto, “A real-time music-scene-description system: Predominant-F0 estimation for detecting melody and bass lines in real-world
audio signals,” Speech Commun., vol. 43, no. 4, pp. 311–329, 2004.

FUJIHARA et al.: LYRICSYNCHRONIZER: AUTOMATIC SYNCHRONIZATION SYSTEM[8] M. Bay, A. F. Ehmann, and J. S. Downie, “Evaluation of multiple-F0
estimation and tracking systems,” in Proc. 10th Int. Soc. Music Inf.
Retrieval Conf. (ISMIR’09), 2009, pp. 315–320.
[9] G. E. Poliner, D. P. Ellis, A. F. Ehmann, E. Gómez, S. Streich, and B.
Ong, “Melody transcription from music audio: Approaches and evaluation,” IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 4, pp.
1247–1256, May 2007.
[10] A. L. Berenzweig and D. P. W. Ellis, “Locating singing voice segments within music signals,” in Proc. IEEE Workshop Applicat. Signal
Process. Audio Acoust. (WASPAA), 2001, pp. 119–122.
[11] W.-H. Tsai and H.-M. Wang, “Automatic detection and tracking oftarget singer in multi-singer music recordings,” in Proc. IEEE Int. Conf.
Acoust., Speech, Signal Process. (ICASSP’04), 2004, pp. 221–224.
[12] T. L. Nwe and Y. Wang, “Automatic detection of vocal segments inpopular songs,” in Proc. 5th Int. Conf. Music Inf. Retrieval (ISMIR’04),
2004, pp. 138–145.
[13] N. Otsu, “A threshold selection method from gray-level histograms,”
IEEE Trans. System, Man, Cybern., vol. SMC-9, no. 1, pp. 62–66, Jan.
1979.
[14] X. R. T. Galas, “Generalized functional approximation for source filter
system modeling,” in Proc. 2nd Eur. Conf. Speech Commun. Technol.
(Eurospeech’91), 1991, pp. 1085–1088.
[15] K. Tokuda, T. Kobayashi, and S. Imai, “Adaptive cepstral analysis of
speech,” IEEE Trans. Speech Audio Process., vol. 3, no. 6, pp. 481–489,
Nov. 1995.
[16] T. Kitahara, M. Goto, and H. G. Okuno, “Pitch-dependent identification of musical instrument sounds,” Appl. Intell., vol. 23, no. 3, pp.
267–275, 2005.
[17] H. Kameoka, M. Goto, and S. Sagayama, “Selective amplifier of periodic and non-periodic components in concurrent audio signals with
spectral control envelopes,” 2006, vol. 2006, pp. 77–84, IPSJ SIG Tech.
Rep., no. 90.
[18] R. E. Méliani, “Accurate keyword spotting using strictly lexicalfillers,” in Proc. 1997 IEEE Int. Conf. Acoust., Speech, Signal Process.
(ICASSP’97), 1997, pp. 907–910.
[19] A. S. Manos and V. W. Zue, “A segment-based wordspotter using phonetic filler models,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal
Process. (ICASSP’97), 1997, pp. 899–902.
[20] J. L. Gauvain and C.-H. Lee, “Maximum a posteriori estimation for
multivariate Gaussian mixture observations of Markov chains,” IEEE
Trans. Speech Audio Process., vol. 2, no. 2, pp. 291–298, Apr. 1994.
[21] C. J. Leggetter and P. C. Woodland, “Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov
models,” Comput. Speech Lang., vol. 9, pp. 171–185, 1995.
[22] S. B. Davis and P. Mermelstein, “Comparison of parametric representation for monosyllabic word recognition,” IEEE Trans. Acoustic,
Speech, Signal Process., vol. ASSP-28, no. 4, pp. 357–366, Aug. 1980.
[23] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka, “RWC music database: Popular, classical, and jazz music databases,” in Proc. 3rd Int.
Conf. Music Inf. Retrieval (ISMIR’02), Oct. 2002, pp. 287–288.
[24] T. Kawahara, A. Lee, K. Takeda, and K. Shikano, “Recent progressof open-source LVCSR engine Julius and Japanese model repository—Software of continuous speech recognition consortium—,” inProc. 6th Int. Conf. Spoken Lang. Process. (Interspeech’04 ICSLP),
2004.
[25] T. Kudo, K. Yamamoto, and Y. Matsumoto, “Applying conditionalrandom fields to Japanese morphological analysis,” in Proc. Conf. Empirical Methods in Natural Lang. Process., 2004, pp. 230–237.
[26] A. Sasou, M. Goto, S. Hayamizu, and K. Tanaka, “An auto-regressive,
non-stationary excited signal parameter estimation method and an evaluation of a singing-voice recognition,” in Proc. IEEE Int. Conf. Acoust.,
Speech, Signal Process. (ICASSP’05), 2005, pp. I-237–I-240.
[27] M. Goto, “Active music listening interfaces based on signal processing,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process.
(ICASSP’07), 2007, pp. IV-1441–IV-1444.

1261

Hiromasa Fujihara received the Ph.D. degree from
Kyoto University, Kyoto, Japan, in 2010 for his workon computational understanding of singing voices.
He is currently a Research Scientist with the National Institute of Advanced Industrial Science andTechnology (AIST), Tsukuba, Japan. His researchinterests include singing information processing andmusic information retrieval.
Dr. Fujihara was awarded the Yamashita Memorial
Research Award from the Information Processing Society of Japan (IPSJ).

Masataka Goto received the Doctor of Engineering
degree from Waseda University, Tokyo, Japan, in
1998.
He is currently the leader of the Media Interaction Group, National Institute of Advanced Industrial
Science and Technology (AIST), Tsukuba, Japan. He
serves concurrently as a Visiting Professor at the Institute of Statistical Mathematics, an Associate Professor (Cooperative Graduate School Program) in the
Graduate School of Systems and Information Engineering, University of Tsukuba, and a Project Manager of the MITOH Program (the Exploratory IT Human Resources Project)
Youth division by the Information Technology Promotion Agency (IPA).
Dr. Goto received 25 awards over the past 19 years, including the Commendation for Science and Technology by the Minister of MEXT ‘‘Young Scientists’Prize,’’ the DoCoMo Mobile Science Awards ‘‘Excellence Award in Fundamental Science,’’ the IPSJ Nagao Special Researcher Award, and the IPSJ Best
Paper Award.

Jun Ogata received the B.E., M.E., and Ph.D. degrees in electronic and information engineering from
Ryukoku University, Kyoto, Japan, in 1998, 2000,
and 2003, respectively.
He is currently a Research Scientist with the National Institute of Advanced Industrial Science andTechnology (AIST), Tsukuba, Japan. His research interests include automatic speech recognition, spoken
language understanding, and speech-based interface.

Hiroshi G. Okuno (M’03–SM’06) received the B.A.
and Ph.D. degrees from the University of Tokyo,
Tokyo, Japan, in 1972 and 1996, respectively.
He worked for NTT, JST, and Tokyo University of
Science. He is currently a Professor of the Graduate
School of Informatics, Kyoto University, Kyoto,
Japan. He was a Visiting Scholar at Stanford University, Stanford, CA, from 1986 to 1988. He hasdone research in programming languages, parallel
processing, and reasoning mechanism in AI. He iscurrently engaged in computational auditory scene
analysis, music scene analysis, and robot audition. He co-edited Computational
Auditory Scene Analysis (Lawrence Erlbaum Associates, 1998), Advanced Lisp
Technology (Taylor & Francis, 2002), and New Trends in Applied Artificial
Intelligence (IEA/AIE) (Springer, 2007).
Prof. Okuno received various awards including the 1990 Best Paper Awardof JSAI, the Best Paper Award of IEA/AIE-2001, 2005, and 2010, IEEE/RSJ
IROS-2010 NTF Award for Entertainment Robots and Systems, and IROS-2001
and 2006 Best Paper Nomination Finalist. He is a member of AAAI, ACM, ASJ,
ISCA, and 5 Japanese societies.