638

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 18, NO. 3, MARCH 2010

A Modeling of Singing Voice Robust to
Accompaniment Sounds and Its Application to Singer
Identification and Vocal-Timbre-Similarity-BasedMusic Information Retrieval
Hiromasa Fujihara, Masataka Goto, Tetsuro Kitahara, Member, IEEE, and Hiroshi G. Okuno, Senior Member, IEEE

Abstract—This paper describes a method of modeling the characteristics of a singing voice from polyphonic musical audio signals
including sounds of various musical instruments. Because singing
voices play an important role in musical pieces with vocals, such
representation is useful for music information retrieval systems.
The main problem in modeling the characteristics of a singing
voice is the negative influences caused by accompaniment sounds.
To solve this problem, we developed two methods, accompanimentsound reduction and reliable frame selection. The former makes
it possible to calculate feature vectors that represent a spectralenvelope of a singing voice after reducing accompaniment sounds.
It first extracts the harmonic components of the predominant
melody from sound mixtures and then resynthesizes the melody byusing a sinusoidal model driven by these components. The lattermethod then estimates the reliability of frame of the obtained
melody (i.e., the influence of accompaniment sound) by using two
Gaussian mixture models (GMMs) for vocal and nonvocal frames
to select the reliable vocal portions of musical pieces. Finally, eachsong is represented by its GMM consisting of the reliable frames.
This new representation of the singing voice is demonstrated to
improve the performance of an automatic singer identification
system and to achieve an MIR system based on vocal timbre
similarity.
Index Terms—Music information retrieval (MIR), singer identification, singing voice, vocal, vocal timbre similarity.

I. INTRODUCTION
HE singing voice is known to be the oldest musical instrument that most people have by nature and plays an important role in many musical genres, especially in popular music.
When a song is heard, for example, most people use the vocals
by the lead singer as a primary cue for recognizing the song.
Therefore, most music stores classify music according to theT

Manuscript received January 01, 2009; revised November 27, 2009. Current
version published February 10, 2010. This work was supported in part by CrestMuse, in part by CREST, JST. The associate editor coordinating the review ofthis manuscript and approving it for publication was Dr. Bertrand David.
H. Fujihara is with the National Institute of Advanced Industrial Science andTechnology (AIST), Tsukuba 305-8568, Japan, and Kyoto University, Kyoto
606-8501, Japan (e-mail: h.fujihara@aist.go.jp).
M. Goto is with the National Institute of Advanced Industrial Science andTechnology (AIST), Tsukuba 305-8568, Japan (e-mail: m.goto@aist.go.jp).
T. Kitahara is with Kwansei Gakuin University, Hyogo 662-8501, Japan(e-mail: t.kitahara@ksc.kwansei.ac.jp).
H. G. Okuno is with Kyoto University, Kyoto 606-8501, Japan (e-mail:okuno@i.kyoto-u.ac.jp).
Digital Object Identifier 10.1109/TASL.2010.2041386

singers’ names (often referred to as artists’ names) in addition
to musical genres.
As the singing voice is important, the representation of its
characteristics is useful for music information retrieval (MIR).
For example, if the name of a singer can be identified without
any information of the metadata of songs, users can find songs
sung by a certain singer using a description of singers’ names(artists’ names). Most previous MIR systems based on metadata, however, have assumed that the metadata including artists’names and song titles were available: if they were not available
for some songs, these songs could not be retrieved by submitting a query of their artists’ names. Furthermore, detailed descriptions of the acoustical characteristics of singing voices canalso play an important role in MIR systems because they areuseful for systems based on vocal timbre similarity by computing acoustical similarities between singers. Hence, a user candiscover new songs rendered by the singing voices they prefer.
To identify singers’ name and compute similarities between
singers without requiring the metadata for each song to be prepared, in this paper, we focused on the problem of representing
the characteristics of the singing voice. This problem was difficult to solve because most singing voices are accompanied byother musical instruments and the feature vectors extracted frommusical audio signals are influenced by the sounds of accompanying instruments. It is therefore necessary to focus on the vocals in polyphonic sound mixtures while considering the negative influences from accompaniment sounds.
We propose two methods of solving this problem: accompaniment sound reduction and reliable frame selection. Using theformer, we can reduce the influence of instrumental accompaniment. We first extracted the harmonic structure of the melody
from audio signals, and then, resynthesized it using a sinusoidal
model. This method reduces the influence of accompanimentsounds. The latter method is used to select reliable frames that
represent the characteristics of the singing voice. We also applied these techniques and implemented an automatic singer
identification system and an MIR system based on vocal timbre
similarity.
II. RELATED STUDIESThe novelty of this paper compared to the previous singer
identification methods lies in our two methods that solve theproblem of the accompaniment sounds. Tsai et al. [1], [2] have1558-7916/$26.00 © 2010 IEEE

FUJIHARA et al.: MODELING OF SINGING VOICE ROBUST TO ACCOMPANIMENT SOUNDS

pointed out the problem of negative influences caused by the accompaniment sounds and have tried to solve it by using a statistically based speaker-identification method for speech signals innoisy environments [3]. On the assumption that singing voices
and accompaniment sounds are statistically independent, theyfirst estimated an accompaniment-only model from interludesections and a vocal-plus-accompaniment model from wholesongs, and then estimated a vocal-only model by subtracting
the accompaniment-only model from the vocal-plus-accompaniment model. However, this assumption is not always satisfied
and the way of estimating the accompaniment-only model has
a problem, i.e., accompaniments during vocal sections and performances (accompaniments) during interlude sections can havedifferent acoustical characteristics. Although Mesaros et al. [4]have tried to solve this problem by using a vocal separation
method similar to our accompaniment sound reduction method,
their method did not deal with the existence of interlude sections where singing voice does not exist and they conducted experiments using the data containing only vocal sections. In otherprevious studies [5]–[9], the accompaniment sound problem has
not been explicitly dealt with.
From the view point of content-based MIR studies, this paper
is important because our system enables a user to retrieve asong based on the specific content of the music. We consideredthat there can be various ways of expressing the content of themusic and it is practical for users to retrieve songs using similarities based on various aspects of the music. Although somestudies [10], [11] attempted to develop MIR systems based on
baseline similarity and instrument existence, most previous content-based MIR systems used low-level acoustic features suchas the MFCCs, the sectral centroid, and rolloff and can retrieve
songs based on only vague similarities [12]–[21]. Pampalk [18]pointed out such limitation of the low-level acoustic features andit is demanded to discover new features and similarity measuresthat can represent more detailed content of the music.
III. REPRESENTATION OF SINGING VOICE ROBUSTTO ACCOMPANIMENT SOUNDS
The main difficulty in modeling the characteristics of asinging voice in polyphonic music lies in the negative influences of accompaniment sounds. Since singing voice isusually accompanied by musical instruments, the acousticalfeatures that are directly extracted from the singing voice will
depend on the accompaniment sounds. When such features as
cepstral coefficients or linear prediction coefficients (LPC) are
extracted, which are commonly used in music-modeling andspeech-modeling studies, those obtained from musical audio
signals will not solely represent the singing voice but a mixtureof the singing voice and the accompaniment sounds. Therefore,
it is essential to cope with this accompaniment sound problem.
One possible solution to this problem is to use data influenced by accompaniment sounds for both training and identification. In fact, most of the previous studies [5]–[8] adopted
this approach. However, this often fails because accompanimentsounds usually have different acoustical features from song tosong. For example, the acoustics between two musical pieces
that are accompanied by a piano solo and a full band will not be
sufficiently similar, even if they are sung by the same singer.

639

Fig. 1. Overview of our method.

We propose a method that can reduce the negative influenceof accompaniment sounds directly from a given musical audio
signal to solve this problem. This feature vector represents vocal
characteristics better than features vector like MFCCs that only
represents a mixture of accompaniment sounds and the singing
voice.
This method consists of the following four parts: accompaniment sound reduction, feature extraction, reliable frame selection, and stochastic modeling. To reduce the negative influenceof accompaniment sounds, the accompaniment sound reduction
part first segregates and resynthesizes the singing voice frompolyphonic audio signals on the basis of its harmonic structure.
The feature extraction part then calculates the feature vectors
from the segregated singing voice. The reliable frame selection
part chooses reliable vocal regions (frames) from the feature
vectors and removes unreliable regions that do not contain vocals or are greatly influenced by accompaniment sounds. The
stochastic modeling part represents the selected features as parameters of the Gaussian mixture model (GMM). Fig. 1 shows
an overview of this method.
A. Accompaniment Sound Reduction
For the accompaniment sound reduction part, we used a
melody resynthesis technique that consisted of the following
three steps:
1) estimating the fundamental frequency (F0) of the vocal
melody using Goto’s PreFEst [22];2) extracting the harmonic structure corresponding to the
melody;3) resynthesizing the audio signal corresponding to the
melody using sinusoidal synthesis.

640IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 18, NO. 3, MARCH 2010

1) F0 Estimation: We used Goto’s PreFEst [22] to estimate
the F0 of the melody line. PreFEst can estimate the most predominant F0 in frequency-range-limited sound mixtures. Since
the melody line tends to have the most predominant harmonic
structure in middle- and high-frequency regions, we can estimate the F0 of the melody line by applying PreFEst with adequate frequency-range limitations.
The following is a summary of PreFEst. After this, is thelog-scale frequency denoted in units of cents (a musical-intervalmeasurement), and is discrete time. Although a cent originally
represented a tone interval (relative pitch), we use it as a unit of
Hz as a criterion, according
absolute pitch using
to Goto [22]. The conversion from hertz to cent is expressed as
(1)
where
represents frequency in cents andrepresents it
in hertz.
Given the power spectrum,, where denotes frequency in cents anddenotes frame number, we first apply
a bandpass filter (BPF) that was designed so that it would
cover most of the dominant harmonics of typical melody
lines. The filtered frequency components can be representedas, whereis the BPF’s frequency
response to the melody line. In this paper, we designed the BPF
according to Goto’s specifications [22]. To make it possible toapply statistical methods, we represent each of the bandpass-filtered frequency components as a probability density function(pdf), called an observed pdf,
(2)Then, we deem each observed pdf to have been generated froma weighted-mixture model of the tone models of all the possible
F0s, which is represented as
(3)
(4)
where
is the pdf of the tone model for each F0, and Fhand Fl are defined as the lower and upper limits of the possible(allowable) F0 range, andis the weight of a tone modelthat satisfies

component and harmonic components. For each component, weallow cent error and extract the local maximum amplitude in
and amplitudeof thethe allowed area. The frequency
th overtone
at timecan be represented as

(6)
(7)
where
denotes the complex spectrum, anddenotes
F0 estimated by the PreFEst. In our experiments, we set to 20.
3) Resynthesis: Finally, we use a sinusoidal model to resynthesize the audio signal of the melody by using the extracted
harmonic structure,and. Changes in phase are approximated using a quadratic function so that the frequency canchange linearly. Changes in amplitude are also approximatedusing a linear function. Hereafter, represents continuous timein units of seconds and represents the duration between two
consecutive frames in units of seconds. The resynthesized audio
signals,, are expressed as
(8)
(9)
(10)

(11)

(12)
(13)
where
is the largest integer not greater than . Note that
represents a (discrete) frame number where the signal at time
belongs and represents relative time from the beginning of
the frame.
4) Evaluation: To evaluate accompaniment sound reduction,
we calculated a difference in the average spectral distortion (SD)between original signals and segregated signals. Given the spectrum of a vocal-only signal, an original polyphonic signal, and a segregated signal, we define the difference of
the average SD by using the following equation:

(5)

(14)

A tone model represents a typical harmonic structure and indicates where the harmonics of the F0 tend to occur. Then, we
estimateusing an EM algorithm and regard it as theF0’s pdf. Finally, we track the dominant peak trajectory of F0s
fromusing a multiple agent architecture.
2) Harmonic Structure Extraction: By using the estimatedF0, we then extract the amplitude of the fundamental frequency

wheredenotes the SD (in dB) of 2 spectraand, denotes the total number of frames that include a singing
voice, and denotes frame number. The difference in the average SD of 40 songs used in the experiments in Section IV-B
was 4.77 dB on average. Note that the vocal-only signals are
obtained from the multitrack data of these songs. This value represents the harmonic component of the accompaniment soundFUJIHARA et al.: MODELING OF SINGING VOICE ROBUST TO ACCOMPANIMENT SOUNDS

641

Fig. 2. Example harmonic structure extraction. (a) An original spectrum and its
envelope. (b) An extracted spectrum and its envelope. (c) A spectrum of vocalonly signal and its envelope.

that is reduced by our method, and indicates that this method
functions effectively.
Fig. 2 shows an example of the harmonic structure extraction.
Fig. 2(a)–(c) shows an original spectrum and its envelope, an extracted spectrum and its envelope, and a spectrum of vocal-only
data and its envelope, respectively. The envelopes were calculated by using the linear prediction coding (LPC). As seen in the
figures, a spectral envelope of extracted spectrum precisely represents formants of singing voice, compared with that of original
spectrum.
To clarify the effectiveness of accompaniment sound reduction, we show a spectrogram of polyphonic musical audio signals, that of the audio signals segregated by the accompaniment sound reduction method, and that of original (groundtruth) vocal-only signals in Fig. 3. It can be seen that harmoniccomponents of accompaniment sound are decreased by executing the accompaniment sound reduction method. Note thatsome errors of F0 estimation that can be seen in the figure will
be removed by after-mentioned reliable frame selection method.

Fig. 3. Example of accompaniment sound reduction. (a) A spectrogram ofpolyphonic signals. (b) A spectrogram of segregated signals. (c) A spectrogramof vocal-only signals.

on the LPC spectrum plays a role of orthogonalization and is
known to be effective in pattern recognition.
2) F0s: We use
which represent the dynamics ofF0’s trajectory, because a singing voice tends to have temporalvariations in its F0 as a consequence of vibrato and such temporal information is expected to express the singer’s characteristics.

B. Feature Extraction

C. Reliable Frame Selection

We calculate feature vectors consisting of two features, fromthe resynthesized audio signals.
1) LPC-Derived Mel Cepstral Coefficients (LPMCCs): It is
known that the individual characteristics of speech signals are
expressed in their spectral envelopes. LPMCCs are mel-cepstralcoefficients of a LPC spectrum [23], [24], which is the method
to estimate the transfer function of vocal tract. Cepstral analysis

Because the F0 of the melody is simply estimated as the most
predominant F0 in each frame [22], the resynthesized audio signals may contain both vocal sound in singing sections and other
instrument sounds in interlude sections. The feature vectors obtained from them therefore include unreliable regions (frames)
where other accompaniment sounds are predominant. The reliable frame selection part removes such unreliable regions and642IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 18, NO. 3, MARCH 2010

TABLE ITRAINING DATA FOR RELIABLE FRAME SELECTIONTABLE IISONGS USED FOR EVALUATION. NUMBERS IN TABLE
ARE PIECE NUMBERS IN RWC-MDB-P-2001

makes it possible to use only the reliable regions for modelingthe singing voice.
1) Procedure: To achieve this, we introduce two kinds of
GMMs, a vocal GMM
and a nonvocal GMM. The
vocal GMM
is trained on feature vectors extracted from thesinging sections, and the nonvocal GMM
is trained on those
extracted from the interlude sections. Given a feature vector, the likelihoods for the two GMMs,
and,correspond to how the feature vector is like a vocal or a(nonvocal) instrument, respectively. We therefore determine
whether the feature vector is reliable or not by using thefollowing equation:

Fig. 4. Precision rate and recall rate of reliable frame selection.

(15)

where is a threshold.
It is difficult to determine a universal constant threshold fora variety of songs because if the threshold is too high for somesongs, there are too few reliable frames to appropriately calculate the similarities. We therefore determine the threshold thatis dependent on songs so that theof all the frames in eachsong are selected as reliable frames. Note that most of the nonvocal frames are rejected in this selection step.
2) Evaluation: We evaluated the reliable frame selection
method by conducting experiments to confirm the following
two facts: 1) the method can reject nonvocal frames and 2)
the method can select frames which are less influenced by
the accompaniment sound. We trained GMM for vocal andnonvocal using songs listed in Table I and used the 40 songs
listed in Table II for evaluation. All of these data are the sameas those used in the experiments in Section IV-B. First, toconfirm 1), we evaluated a precision rate and a recall rate of
the method and Fig. 4 shows the results. When is 0.15, theprecision rate is approximately 79% and, thus, we can confirm
that many nonvocal sections are rejected. Then, to confirm 2),
Fig. 5 shows a dependency on of spectral distortion of framesthat are selected by reliable frame selection. The average SD ispositively correlated with . Therefore, we could confirm that

Fig. 5. Dependency of spectral distortion of selected frames on .

the method can select frames that are less influenced by theaccompaniment sound, by setting to a smaller value.
D. Stochastic Modeling
Finally, we model a probability distribution of the feature vectors for a song using GMM and estimate the parameters of the
GMM with the EM algorithm. In our experiments, we set the
number of Gaussians to 64.
IV. SINGER IDENTIFICATIONThis section describes one of the applications of our vocal
modeling techniques, i.e., the system for identifying the singer
by determining a singer’s name from given musical audio signals. The target data are real-world musical audio signals suchas popular music CD recordings that contain the singing voicesof single singers and accompaniment sounds.

FUJIHARA et al.: MODELING OF SINGING VOICE ROBUST TO ACCOMPANIMENT SOUNDS

643

A. Determination of Singer
First, we prepare the audio signals of target singers as training
data and calculate the GMMs for all singers by using the method
described in Section III. Given input audio signals, we also calculate the GMM for the song. Then, the name of the singer isdetermined through the following equation:(16)

B. Experiments Using RWC Music Database
We conducted experiments to evaluate our singer identification system.
1) Condition and Results: We conducted experiments on
singer identification using the “RWC Music Database: PopularMusic (RWC-MDB-P-2001)” [25] under the following four
conditions to find out how effective our methods of accompaniment sound reduction and reliable frame selection were as
follows:1) without either reduction or selection (baseline);2) with reduction, without selection;3) without reduction, but with selection;4) with both reduction and selection (ours).
We used 40 songs by ten different singers (five were males
and five were females), listed in Table II, taken from the
RWC-MDB-P-2001. Using these data, we conducted the fourfold cross validation, that is, we first divided all the data into
four groups,() in Table II, and then repeatedthe following step four times; each time, we left out one of
the four groups for training and used the one we had omitted
for testing. We used 25 songs of 16 different singers listed inTable I, also taken from the RWC-MDB-P-2001, which differ
from the singers used for evaluation, as the training data for the
reliable frame selection. We set to 15%, using the experimentdescribed in Section IV-B2 as a reference. To evaluate theperformance of the LPMCCs, we use both the LPMCCs and the
MFCCs as the feature vectors and compare the result. Acuracy
was defined by a ratio of the number of correctly identified
song to the number of songs used for evaluation.
Fig. 6 shows the results of the experiments. As seen in thetable, accompaniment sound reduction and reliable frame selection improved the accuracy of singer identification. When these
two methods were used together, in particular, the accuracy wassignificantly improved from 55% to 95%.
Fig. 7 shows the confusion matrices of the experiments whenthe LPMCCs are used. As can be seen, confusion between
males and females decreased by using the reduction method.
This means that, under conditions 2) and 4), the reduction
method decreased the influence of accompaniment sound,and the system could correctly identify the genders. However,
without the reduction method [conditions 1) and 3)], the influences of accompaniment sound prevented the system fromcorrectly identifying even the genders of the singers.
When we compare the MFCCs and the LPMCCs, we can findthat the accuracies of the LPMCCs exceed those of the MFCCs
in all the conditions. This is particularly remarkable when we

Fig. 6. Results of the experiments using RWC Music Database, where “reduc.”and “selec.” correspond to accompaniment sound reduction and reliable frame
selection, respectively.

use both the accompaniment sound reduction method and the reliable frame selection method. We can confirm that the LPMCCs
represent the characteristics of the singing voice well.
2) Dependence of Accuracy on : We conducted experiments by setting to various values to investigate how dependent the accuracies were on , which represents the percentage
of frames determined to be reliable by using the reliable frame
selection method. These experiments used the same dataset asthat used in the previous experiments. The experimental resultsin Fig. 8 indicate that the accuracy of classification was not affected by small changes in . We can also see that the valueof that yielded the highest accuracy differed. The reason forthis is as the follow: accompaniment sound reduction method reduced the influence of accompaniment sounds and emphasized
the differences between reliable and unreliable frames. Thus, if
we increased excessively, the system selected many unreliable
frames and the performance of the system decreased.
3) Combination of Accompaniment Sound Reduction and Reliable Frame Selection: To confirm an effectiveness of reliable frame selection in combination with the accompanimentsound reduction method, we conducted experiments under thefollowing three conditions.
1) Only hand-labeled vocal sections are used. We executeaccompaniment sound reduction using ground-truth
F0s. We do not execute reliable frame selection.
2) Only hand-labeled vocal sections are used. We executeaccompaniment sound reduction using F0s estimated by
Prefest. We do not execute reliable frame selection.
3) An entire region of a song are used. We execute accompaniment sound reduction using F0s estimated by
PreFEst and reliable frame selection.
Table III shows the results of the experiments. When we compare condition 3) with condition 2), the accuracy was improvedby 12 points (from 83% to 95%). This fact indicates that the reliable frame selection method can achieve higher accuracy thanthe manual removal of nonvocal sections. When we compare
condition 3) with condition 1), the accuracy was improved by 7
points (88% to 95%). This fact indicates that, even if there were
no F0 estimation errors, it was difficult to achieve high accuracy without reliable frame selection. In contrast, this fact also
indicates that some F0 estimation errors did not degrade system
performance if we use a reliable frame selection method because

644IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 18, NO. 3, MARCH 2010

Fig. 7. Confusion matrices. Center lines in each figure are boundaries between males and females. Note that confusion between males and females decreased byusing the accompaniment sound reduction method.

TABLE IVARTISTS SELECTED FROM COMMERCIAL CD RECORDINGS

Fig. 8. Experimental results for dependence of accuracy on .
was determined to be reliable.

% of all frames

TABLE IIIEVALUATION OF A COMBINATION OF ACCOMPANIMENT SOUND
REDUCTION AND RELIABLE FRAME SELECTION, WHERE “SELEC.”
MEANS RELIABLE FRAME SELECTION

the method rejected the region in which PreFEst failed to estimate correct F0s.

TABLE VQUERY SONGS AND RETRIEVED CORRESPONDING SONGS USED FOR
SUBJECTIVE EXPERIMENT: THREE-DIGIT NUMBER INDICATES THE PIECENUMBER OF THE RWC-MDB-P-2001. GIVEN EACH QUERY SONG,
TOP-RANKED SONG BY BASELINE METHOD (MFCC) AND TOP-RANKEDSONG BY OUR METHOD ARE SHOWN ON SAME LINE

C. Experiments Using Commercial CD Recordings
We also conducted experiments using commercial CD
recordings available in Japan. The experiments were done
under the four conditions described in Section IV-B. 246 songsby 20 singers (8 males and 12 females) listed in Table IV wereused in these experiments. These artists were selected fromthe Japanese best-seller list of CD in 2004. The same as in
the previous experiments, the 25 songs by 16 singers listed
in Table I were used as the training data for reliable frame
selection. Using these data, we conducted the fourfold crossvalidation.
The bar chart in Fig. 9 shows the results of these experiments. We confirmed that the accuracy improved by approximately 12% by using both methods, while accuracy improved
by approximately 8% by using each of the two methods.
Fig. 10 shows the confusion matrices. As can be seen, thesystem more often misidentified songs by female singers thanthose by males. We consider this is because the pitch of female
singing is generally higher than that of male singing. We found
spectral envelopes estimated from high-pitched sounds by using
cepstrum or LPC analysis are strongly affected by spectral valleys between adjacent harmonic components.

FUJIHARA et al.: MODELING OF SINGING VOICE ROBUST TO ACCOMPANIMENT SOUNDS

645

and
correspond to the GMM parametersY, respectively,of songs X and Y, respectively, andrepresents thelikelihood of GMM with parameter .
B. System Operation

Fig. 9. Results of the experiments using commercial CD-recordings, where“reduc.” and “selec.” mean accompaniment sound reduction and reliable frame
selection, respectively.

The accuracy of the baseline method (Condition 1) in the experiments using the commercial CDs was higher than that of the
RWC Music Database by approximately 15%. This is because
songs on the same album tend to use the same instruments andbe homogenous in sound quality. Berenzweig et al. [6] called
this phenomenon the “Album effect” and they pointed out thatthe performance of a singer identification system depends on
what kind of dataset is used. On the other hand, since the RWCMusic Database consists of a variety of genres and instrumentseven for songs by the same singer, the accuracy of the baseline method was only 55%. However, since the proposed method(Condition 4) was extremely accurate for the experiments using
the RWC Music Database, we found that our method can identify the singers’ names correctly even if there were a variety ofsongs in the database.
V. MIR BASED ON VOCAL TIMBRE SIMILARITY
We also applied our technique to a new MIR system basedon vocal timbre similarity and developed a system named VocalFinder. In this paper, the term vocal timbre means a shape ofa spectral envelope of the singing voice. By using this system,
we could find a song by using its musical content in addition
to traditional bibliographic information. This kind of retrieval
is called content-based MIR, and our system, which focuses on
singing voices as content, falls into this category.
A. Similarity Calculation
We chose symmetric Kullback–Leibler divergence [26] to be
the similarity measure between two songs. Since it is difficult
to calculate this similarity measure in a closed form, we approximate it as follows (this approximation is called “cross-likelihood ratio test”[26]); the similarity between songs X and Y
is calculated by

(17)
and
correspond to the feature vectors of reliable
whereframes, which could be MFCCs or LPMCCs, in songs X and

Fig. 11 shows a screenshot of the system. As the training data
for the vocal and nonvocal GMMs, we used the same 25 songs
listed and used in the experiences in Section IV-B. We registered
the other 75 songs from the RWC-MDB-P-2001 in the system
database, which were not used to construct these GMMs. In
the figure, the song “PROLOGUE” (RWC-MDB-P-2001 No.7)
sung by the female singer “Tomomi Ogata” is given as a query.
Given a query song, it took about 20 seconds to calculate similarities and output a ranked list of retrieved songs. As seen in
the Fig. 11, the retrieval results list the ranking, the song titles,
the artists’ names, and similarities.
In most of songs retrieved given various queries, the vocal
timbres of the top ten songs were generally similar to that ofeach query song in our experience. For example, in Fig. 11, thetop 21 songs were sung by female singers, and the vocal timbresof the top 15 songs in this figure were similar to the query song.
Note that four songs by “Tomomi Ogata” who was the singer
in the query took first, second, ninth, and twelfth places. This isbecause the singing styles of the ninth and twelfth songs weredifferent from those of the first and second songs and the query.
C. Subjective Experiment
We conducted a subjective experiment to compare our systemusing the proposed vocal-based feature vector with a baseline
system using the traditional MFCCs of the input sound mixtures.
Six university students (two males and four females) participated in this experiment. They had not received any professional training in music. They first listened to a set of threesongs—a query song (song X), the top-ranked song retrievedby our system (songs A/B), and the top-ranked song retrievedby the baseline system (songs B/A)—, and then judged whichsong was more similar to the query song (Fig. 12). They did
not know which song was retrieved by our system and the songorder of A and B was randomized. We allowed them to listen tothese songs in any order for as long as they liked.
We selected ten query songs from the system database taking
into consideration that these songs were sung by different genders and in different genres. For each query song, we asked thesubjects the following questions.
• Question 1: When comparing the singing voice timbres ofsongs A and B, which song resembles song X?
• Question 2: When comparing the overall timbres of songs
A and B, which song resembles song X?
Figs. 13 and 14 show the results of the experiment. On average, 80% of the responses for ten songs judged that the timbreof the singing voice obtained by our method was more similar to that of the query song (Fig. 13). A binominal test, inwhich significance was set at 0.05, was performed on these results and the degree of significance was 0.0000086, which indicates that there were significant differences between our method
and the conventional method. On the other hand, 70% of the responses judged that the overall timbre obtained by the baseline
method was more similar to that of the query song (Fig. 14).

646

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 18, NO. 3, MARCH 2010

Fig. 10. Confusion matrices of experiments using commercial CD-recordings. Center lines in each figure are boundaries between males and females.

Fig. 13. Evaluation results: Question 1: singing voice timbre.

Fig. 11. Screenshot of the system.

Fig. 14. Evaluation results: Question 2: overall timbre.

subjects judged that the song obtained by our method was moresimilar to the query song in terms of the vocal timbre similarity.
Fig. 12. Interface used for subjective experiment.

We also preformed a binominal test and the degree of significance was 0.033. Therefore, we confirmed that our method can
reduce the influence of accompaniment sounds and find songsby using vocal timbres. We also found that our method finds
not only songs with similar vocal timbres (or by same singer)
but also songs with similar singing styles. For example, whensong RWC-MDB-P-2001 No.53 was used as a query, both our
method and the baseline method retrieved the top-ranked songsby the singer to be the same as that in the query, but 5 out of 6

VI. DISCUSSION
This section discusses the novelty and effectiveness of themethod proposed in this paper.
A. Novelty and Effectiveness of Accompaniment Sound
Reduction
We clarified the problem caused by accompaniments whenmodeling the singing voice, which has not effectively been
dealt with except for a few attempts. We provided two effective
solutions, i.e., accompaniment sound reduction and reliable
frame selection. The accompaniment sound reduction method

FUJIHARA et al.: MODELING OF SINGING VOICE ROBUST TO ACCOMPANIMENT SOUNDS

is characterized by the way it dealt with accompaniment sound:
it segregated the singing voice directly from the spectrum of
the singing voice without modeling the accompaniment sound.
Although the conventional method dealt with this problem bymodeling the accompaniment sound, it was generally difficult
to model the accompaniment sound.
In this paper, we conducted two disparate experiments to confirm the effectiveness of this method. First, we evaluated thedifference in the average spectral distortion and found that themethod reduced the spectral distortion by 4.77 dB. Second, theresults of the experiments on singer identification showed thatthe method improved identification accuracy from 70% to 95%
for the data taken from the RWC Music Database and from
88.6% to 95.3% for the data taken from commercial CD recordings.
B. Novelty and Effectiveness of Accompaniment Sound
Reduction
The reliable frame selection method made it possible to consistently select reliable frames that represented the characteristics of the singing voice. It needs to be noted that this method
even rejected unreliable vocal frames as well as nonvocal frames
to improve the robustness. Although similar methods were usedin previous studies, they focused on distinguishing vocal andnonvocal frames; they did not consider the reliability of eachframe.
The effectiveness of the reliable frame selection method is
confirmed by the following two comparisons. First, we compared the spectral distortion between frames selected and thoserejected by the method. The spectral distortion of the former
was smaller than the letter, and therefore, we could say thatthe method can select frames that are less influenced by the accompaniment sounds. Then, we compared the results of experiments on signer identification. The accuracy of the experimentin which the hand-labeled vocal sections are used without the selection method was 83%, while that with the selection method
was 95%. This result indicates that it is important not only todetect the vocal regions but also to select reliable frames.
C. Effectiveness of a Combination of the Two Methods
It needs to be noted that the reliable frame selection method
is robust to the error of the accompaniment sound reduction
method because the reliable frame selection method can rejectframes in which the singing voice was not properly segregatedby the accompaniment sound reduction method. This is confirmed by the experiments using the ground-truth F0 for the accompaniment sound reduction method. Though methods similar to the accompaniment sound reduction have been used to
improve the noise robustness in the field of speech recognition[27], this is the first paper that proposed a method that can beused in combination with the accompaniment sound reduction
method and increase robustness to F0 estimation errors.
VII. CONCLUSION
We described two methods that work in combination tomodel the characteristics of the singing voice. To deal withthe singing voice including sound mixtures of various musical647

instruments, our method solved the problem of the accompaniment sound influences. We developed an automatic singer
identification system and an MIR system based on vocal timbre
similarity by applying the new representation of the singing
voice, and tested and confirmed the effectiveness of thesesystems by conducting objective and subjective experiments.
In the future, we plan to extend our method to represent
singing styles of singers in addition to the vocal timbre bymodeling F0’s trajectories of the singing voices. We also plan
to integrate this system with content-based MIR methods basedon other musical elements to give users a wider variety of
retrieval methods.
REFERENCES[1] W.-H. Tsai and H.-M. Wang, “Automatic detection and tracking oftarget singer in multi-singer music recordings,” in Proc. 2004 IEEE
Int. Conf. Acoust., Speech, Signal Process. (ICASSP 2004), 2004, pp.
221–224.
[2] W.-H. Tsai and H.-M. Wang, “Automatic singer recognition of popular
music recordings via estimation and modeling of solo vocal signals,”IEEE Trans. Audio, Speech, Lang. Process., vol. 14, no. 1, pp. 330–341,
Jan. 2007.
[3] R. C. Rose, E. M. Hofstetter, and D. A. Reynolds, “Integrated modelsof signal and background with application to speaker identification innoise,” IEEE Trans. Speech Audio Process., vol. 2, no. 2, pp. 245–257,
Mar. 1994.
[4] A. Mesaros, T. Virtanen, and A. Klapuri, “Singer identification in polyphonic music using vocal separation and pattern recognition methods,”
in Proc. 8th Int. Conf. Music Inf. Retrieval (ISMIR 2007), 2007, pp.
375–378.
[5] B. Whitman, G. Flake, and S. Lawrence, “Artist detection in music withminnowmatch,” in Proc. 2001 IEEE Workshop Neural Netw. Signal
Process., 2001, pp. 559–568.
[6] A. L. Berenzweig, D. P. W. Ellis, and S. Lawrence, “Using voice segments to improve artist classification of music,” in Proc. AES 22nd Int.
Conf. Virtual, Synth., Entertainment Audio, 2002.
[7] Y. E. Kim and B. Whitman, “Singer identificatin in popular music
recordings using voice coding features,” in Proc. 3rd Int. Conf. Music
Inf. Retrieval (ISMIR2002), 2002, pp. 164–169.
[8] T. Zhang, “Automatic singer identification,” in Proc. IEEE Int. Conf.
Multimedia Expo (ICME 2003), 2003, vol. I, pp. 33–36.
[9] W.-H. Tsai, S.-J. Liao, and C. Lai, “Automatic identification of simultaneous singers in duet recordings,” in Proc. 9th Int. Conf. Music Inf.
Retrieval (ISMIR 2008), 2008, pp. 115–120.
[10] T. Kitahara, M. Goto, K. Komatani, T. Ogata, and H. G. Okuno, “Instrogram: Probabilistic representation of instrument existence for polyphonic music,” IPSJ J., vol. 48, no. 1, pp. 214–226, 2007.
[11] Y. Tsuchihashi, T. Kitahara, and H. Katayose, “Using bass-line features for content-based mir,” in Proc. 9th Int. Conf. Music Inf. Retrieval(ISMIR 2008), 2008, pp. 620–625.
[12] J.-J. Aucouturier and F. Pachet, “Music similarity measures: What’s
the use?,” in Proc. 3rd Int. Conf. Music Inf. Retrieval (ISMIR2002),
2002, pp. 157–163.
[13] B. Logan, “Content-based playlist generation: Exploratory experiments,” in Proc. 3rd Int. Conf. Music Inf. Retrieval (ISMIR 2002),
2002, pp. 295–296.
[14] E. Allamanche, J. Herre, O. Hellmuth, T. Kastner, and C. Ertel, “A
multiple feature model for musical similarity retrieval,” in Proc. 4th
Int. Conf. Music Inf. Retrieval (ISMIR 2003), 2003, pp. 217–218.
[15] A. Berenzweig, B. Logan, D. P. W. Ellis, and B. Whitman, “A largescale evaluation of acoustic and subjective music similarity measures,”in Proc. 4th Int. Conf. Music Inf. Retrieval (ISMIR 2003), 2003, pp.
63–70.
[16] M. F. McKinney and J. Breebaart, “Features for audio and music classification,” in Proc. 4th Int. Conf. Music Inf. Retrieval (ISMIR 2003),
2003, pp. 151–158.
[17] G. Tzanetakis, J. Gao, and P. Steenkiste, “A scalable peer-to-peer
system for music content and information retrieval,” in Proc. 4th Int.
Conf. Music Inf. Retrieval (ISMIR 2003), 2003, pp. 209–214.
[18] E. Pampalk, “Computational models of music similarity and their application in music information retrieval,” Ph.D. dissertation, Universitat Wien, Vienna, Austria, 2006.

648

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 18, NO. 3, MARCH 2010[19] A. Flexer, F. Gouyou, S. Dixon, and G. Widmer, “Probabilistic combination of features for music classification,” in Proc. 7th Int. Conf.
Music Inf. Retrieval (ISMIR 2006), 2006, pp. 628–633.
[20] T. Pohle, P. Knees, M. Schedl, and G. Widmer, “Independent component analysis for music similarity computation,” in Proc. 7th Int. Conf.
Music Inf. Retrieval (ISMIR 2006), 2006, pp. 228–233.
[21] D. P. W. Ellis, “Clasifying music audio with timbral and chroma features,” in Proc. 8th Int. Conf. Music Inf. Retrieval (ISMIR 2007), 2007,
pp. 339–340.
[22] M. Goto, “A real-time music-scene-description system: Predominant-F0 estimation for detecting melody and bass lines in real-world
audio signals,” Speech Commun., vol. 43, no. 4, pp. 311–329, 2004.
[23] B. S. Atal, “Effectiveness of linear prediction characteristics of the
speech wave for automatic speaker identification and verification,” J.
Acoust. Soc. Amer., vol. 55, no. 6, pp. 1304–1312, 1974.
[24] K. Shikano, Evaluation of LPC spectral matching measures for phonetic unit recognition Comput. Sci. Dept. Carnegie Mellon Univ., Tech.
Rep. CMU-CS-96-108, 1986.
[25] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka, “RWC music database: Popular, classical, and jazz music databases,” in Proc. 3rd Int.
Conf. Music Inf. Retrieval (ISMIR 2002), Oct. 2002, pp. 287–288.
[26] T. Virtanen and M. Helen, “Probabilistic model based similarity
measures for audio query-by-example,” in Proc. 2007 IEEE Workshop
Applicat. Signal Process. Audio Acoust. (WASPAA 2007), 2007, pp.
82–85.
[27] T. Nakatani and H. G. Okuno, “Harmonic sound stream segregation
using localization and its application to speech stream segregation,”
Speech Commun., vol. 27, pp. 209–222, 1999.

Hiromasa Fujihara received the B.S. and M.S. degrees from Kyoto University, Kyoto, Japan, in 2005
and 2007, respectively. He is currently pursuing
the Ph.D. degree in the Department of Intelligence
Science and Technology, Graduate School of Informatics, Kyoto University.
He is currently a Research Scientist of the National Institute of Advanced Industrial Science andTechnology (AIST), Tsukuba, Japan. His researchinterests include singing information processing andmusic information retrieval.
Mr. Fujihara was awarded the Yamashita Memorial Research Award from theInformation Processing Society of Japan (IPSJ).

Masataka Goto received the Doctor of Engineering
degree from Waseda University, Tokyo, Japan, in
1998.
He is currently a Leader of the Media Interaction
Group, Information Technology Research Institute,
National Institute of Advanced Industrial Scienceand Technology (AIST), Tsukuba, Japan. He servesconcurrently as a Visiting Professor in the Department of Statistical Modeling, The Institute of
Statistical Mathematics, and an Associate Professor(Cooperative Graduate School Program) in the Department of Intelligent Interaction Technologies, Graduate School of Systems
and Information Engineering, University of Tsukuba.
Dr. Goto received 24 awards over the past 17 years, including the Commendation for Science and Technology by the Minister of MEXT “Young Scientists’ Prize,” DoCoMo Mobile Science Awards “Excellence Award in Fundamental Science,” IPSJ Nagao Special Researcher Award, and the IPSJ Best
Paper Award.

Tetsuro Kitahara (M’07) received the B.S. degreefrom Tokyo University of Science, Tokyo, Japan, in
2002 and the M.S. and Ph.D. degrees from Kyoto
University, Kyoto, Japan, in 2004 and 2007, respectively.
He is currently a Postdoctoral Researcher at
Kwansei Gakuin University, Hyogo, Japan, for the
CrestMuse Project funded by CREST, JST, Japan.
His research interests include music informatics andcomputational auditory scene analysis.
Dr. Kitahara received several awards including theSecond Kyoto University President Award.

Hiroshi G. Okuno (SM’06) received B.A. and Ph.D.
from the University of Tokyo in 1972 and 1996, respectively.
He worked for NTT, JST, and Tokyo Universityof Science. He is currently a Professor of Graduate
School of Informatics, Kyoto University, Kyoto,
Japan. He was a Visiting Scholar at Stanford University, Stanford, CA, from 1986 to 1988. He hasdone research in programming languages, parallel
processing, and reasoning mechanism in AI. He iscurrently engaged in computational auditory scene
analysis, music scene analysis, and robot audition. He coedited Computational
Auditory Scene Analysis (Lawrence Erlbaum Associates, 1998), Advanced Lisp
Technology (Taylor and Francis, 2002), and New Trends in Applied Artificial
Intelligence (IEA/AIE) (Springer, 2007).
Dr. Okuno received various awards including the 1990 Best Paper Award of
the JSAI, the Best Paper Award of IEA/AIE-2001 and 2005, and the IEEE/RSJ
IROS-2001 and 2006 Best Paper Nomination Finalist. He is a member of the
AAAI, ACM, ASJ, ISCA, and five Japanese societies.