F0 ESTIMATION METHOD FOR SINGING VOICE IN POLYPHONIC AUDIO SIGNALBASED ON STATISTICAL VOCAL MODEL AND VITERBI SEARCH
Hiromasa Fujihara,† Tetsuro Kitahara,† Masataka Goto,‡
Kazunori Komatani,† Tetsuya Ogata,† and Hiroshi G. Okuno††‡

Dept. of Intelligence Science and Technology
Graduate School of Informatics, Kyoto University
Sakyo-ku, Kyoto 606-8501, Japan{fujihara,kitahara,komatani,ogata,okuno}@kuis.kyoto-u.ac.jp

National Institute of Advanced Industrial
Science and Technology (AIST)
Tsukuba, Ibaraki 305-8568, Japanm.goto@aist.go.jp

ABSTRACT

Input (Audio signals)

This paper describes a method for estimating F0s of vocal frompolyphonic audio signals. Because melody is sung by a singer inmany musical pieces, the estimation of F0s of the vocal part is useful for many applications. Based on existing multiple-F0 estimation
method, we evaluate the vocal probabilities of the harmonic structure of each F0 candidate. In order to calculate the vocal probabilities of the harmonic structure, we extract and resynthesize theharmonic structure by using a sinusoidal model and extract feature
vectors. Then, we evaluate the vocal probability by using vocal andnon-vocal Gaussian mixture models (GMMs). Finally, we track F0
trajectories using these probabilities based on Viterbi search. Experimental results show that our method improves estimation accuracy
from 78.1% to 84.3%, which is 28.3% reduction of misestimation.

(a)

time

(b)

freq.

Vocal probability evaluation

time

(c)

freq.

F0 tracking based on Viterbi searchtime

1. INTRODUCTION
Singing voice plays an important role in many musical genres, especially in popular music. The estimation of the fundamental frequency (F0) of the vocal part, therefore, is an important issue. Estimated F0s of vocal are useful in various applications, such as automatic transcription, automatic generation of Karaoke track andmusic information retrieval (e.g., searching for a song by singing
a melody).
Several study have been made on melody extraction from polyphonic audio signals [1, 2, 3, 4]. Their methods usually consisted oftwo steps. First, they estimated F0 candidates from input audio signals using multiple-pitch estimation method. Then, among these F0
candidates, they constructed a melody trajectory by using some clues
such as predominance, timber, meter and F0’s continuity. However,the F0s of other predominant instruments performed concurrently
with vocals were often detected because they did not assume what asound source was [1].
In this paper, we focus on the F0 estimation of vocal from polyphonic audio signals. Polyphonic audio signals contain frequency
components corresponding to various F0s coming from the instruments. Without vocal/non-vocal discrimination focusing on a frequency component corresponding to a particular F0, it is difﬁcult toestimate the F0s of vocal. In previous vocal/non-vocal discrimination methods [5, 6, 7], since feature vectors were extracted directly
This research was supported in part by the Ministry of Education, Culture, Sports, Science and Technology (MEXT), Grant-in-Aid for ScientiﬁcResearch (A), No.15200015, and Informatics Research Center for Development of Knowledge Society Infrastructure (COE program of MEXT, Japan).
We thank everyone who has contributed to building and distributing the RWCMusic Database [11]. We also thank Hirokazu Kameoka, Kazuyoshi Yoshii
and Takuya Yoshioka for their valuable advice.

1­4244­0469­X/06/$20.00 ©2006 IEEE

freq.

Pitch likelihood calculation

Fig. 1. Overview of our method.
from polyphonic audio signals, they did not identify a sound sourceof a particular F0. In order to overcome this technical issue, we use
accompaniment sound reduction we developed [8], which enable us
to accurate F0 estimation focusing on vocal.
There are two key ideas behind our method. First is to evaluate the probability that harmonic structure of each F0 candidate
is vocal, which we call vocal probability, by using vocal and nonvocal GMMs. We enable it by using accompaniment sound reduction. First, we extract and resynthesize the harmonic structure ofeach F0 candidate by using a sinusoidal model and extract feature
vector from the resynthesized audio signal. Then, we evaluate thevocal probability according to the Bayes rule. Second is to track
vocal’s F0 trajectory based on Viterbi search considering these vocal
probabilities. We enable it by modeling and formulating the stochastic dependency among a sequence of F0s, spectra and sound sources
using a graphical model.
2. OUR METHOD FOR ESTIMATING F0 OF VOCAL
Figure 1 shows an overview of our method. It consists of threeparts: pitch likelihood calculation, vocal probability calculation andF0 tracking based on Viterbi search. We deﬁne pitch likelihood as
likelihood that an F0 is the most predominant F0 in a spectrum.
First, we calculate pitch likelihood (Figure 1 (a)) using PreFEst[1]developed by Goto. Given a spectrum, it calculates a predominanceof every possible F0 at each time. We consider the predominanceas pitch likelihood. Then, we evaluate vocal probability for each F0(Figure 1 (b)). Finally, considering the vocal probabilities and theV ­ 253ICASSP 2006

Observed spectrum

for each F0 candidate

Reliable
Frame
Selection

Fl

=

Al

=

|ψ(Fl )|,

(2)

2. Resynthesis
We resynthesize the audio signal of the melody from the extracted harmonic structure by using a sinusoidal model [9].
Resynthesized audio signals s(t) are expressed as

Vocal
GMM

Vocal
probability(1)fwhere ψ denotes the spectrum, f denotes the target F0, andG(x; m, σ) represents the Gaussian distribution.

Harmonic
Structure
Extraction
Resynthesis

Non-Vocal
GMM

argmax |ψ(f )|G(f ; lf , 20),s(t) =

Vocal
probability

LX

Al cos(ωl t).

(3)

l=1

2.2.2. Feature Extraction

Fig. 2. Vocal probability calculation.
continuity of F0s, we estimate the time series of F0s that gives thehighest probability (Figure 1 (c)).
Hereafter, f represents the log-scale frequency denoted in unitsof cents (a musical-interval measurement).

From the resynthesized audio signals, we calculate feature vectors
consisting of two features.
• LPC-derived mel cepstral coefﬁcients (LPMCCs)
We use LPMCCs as spectral feature for vocal/non-vocal discrimination because we have reported that, in the context of
singer identiﬁcation, LPMCCs express vocal’s characteristics better than mel-frequency cepstral coefﬁcients (MFCCs),
which are widely used for music modeling [8].

2.1. Pitch Likelihood
We use Goto’s PreFEst [1] for calculating pitch likelihood. Though
PreFEst consists of three steps, front-end, core and back-end, we do
not use the PreFEst-back-end method here because it selects most
predominant pitch trajectory.
We will describe a summary of PreFEst-core below. Given the
power spectrum ψ(f ), in order to enable the application of statisticalmethods, we represent each of the frequency components as a probability Rdensity function (PDF), called an observed PDF: pψ (f ) =∞ψ(f )/ −∞ ψ(f )df . Then, we consider each observed PDF to havebeen generated from a weighted-mixture model of the tone modelsof all the possible F0s, which is represented as follows: p(f |θ) =R Fhw(F )p(f |F )dF , θ = {w(f )|Fl ≤ f ≤ Fh}, where p(f |F ) isFl
the PDF of the tone model for each F0, and Fh and Fl denote lower
and upper limits of the possible (allowable) F0 range, and w(f ) is
R Fl
the weight of a tone model that satisﬁes Fhi w(f )df = 1. Toneimodel represents a typical harmonic structure and indicates where
the harmonics of the F0 tend to occur. Then, we estimate w(f ) using EM algorithm and regard it as the F0’s PDF. We consider F0’s
PDF estimated by PreFEst-core as pitch likelihood function, that isp(ψ|f ) = w(f ). Considering the computational cost, we select 10
candidates that have the highest likelihoods.
2.2. Vocal/Non-vocal Probability
We describe the method that enables us to calculate feature vectors corresponding to each F0 candidate and its vocal and non-vocal
probabilities. Figure 2 shows an overview of this method.

• ∆F0sWe use ∆F0s [10], which represent the dynamics of F0’s trajectory, because singing voice tends to have temporal variation of F0s in consequence of vibrato and, therefore, ∆F0s
are expected to be good cues for vocal/non-vocal discrimination.
2.2.3. Probability Calculation
We introduce two Gaussian mixture models (GMMs): a vocal GMM,
sV , and a non-vocal GMM, sN . The parameters of the vocal GMM,
θV , is trained on feature vectors extracted from vocal sections, andthe parameters of the non-vocal GMM, θN , is trained on those extracted from interlude sections.
When x(ψt , f ) is the feature vector extracted from a spectrum,
ψt , with a fundamental frequency f at time t, the likelihoods of the
vocal and non-vocal model are represented asp(ψt , f |sV )p(ψt , f |sN )==NGMM (x(ψt , f ); θV ),NGMM (x(ψt , f ); θN ),where NGMM (x; θ) denotes the probability density function of the
GMM with parameter θ.
The vocal and non-vocal probabilities, according to the Bayes
rule, can be represented asp(ψt , f |sV )p(sV ),p(ψt , f |sV )p(sV ) + p(ψt , f |sN )p(sN )p(ψt , f |sN )p(sN )p(sN |ψt , f ) =,p(ψt , f |sV )p(sV ) + p(ψt , f |sN )p(sN )p(sV |ψt , f ) =2.2.1. Accompaniment Sound Reduction
To calculate the feature vector of each of the F0 candidate, we use
accompaniment sound reduction, which we proposed in [8]. By using this method, we can obtain the audio signal that corresponds only
to the F0. This method consists of the following two steps.
1. Harmonic Structure Extraction
We extract the power and the phase of fundamental frequency
component and harmonic components. The extracted power,
Al , and frequency, Fl , of the l-th overtone (l = 1, . . . , 20)
can be represented as

(4)
(5)

(6)
(7)

where p(sV ) and p(sN ) denote a priori probabilities of vocal andnon-vocal. We used 64-mixture GMM and set p(sV ) = p(sN ).
2.3. F0 Tracking Based on Viterbi Search
We formulate the F0 tracking method based on Viterbi search.

V ­ 254

f1

㱤

ft-1

㱤

1

t-1

㱗

t-1

ft+1

㱤

t

㱗

1

㱤

ft

㱤

t+1

㱗

t

㱗

fT

Table 1. Training data for Vocal/Non-Vocal model.
Name
Shingo Katsuta
Yoshinori Hatae
Masaki Kuehara
Hiroshi Sekiya
Katsuyuki Ozawa
Masashi Hashimoto
Satoshi Kumasaka
Konbu
Eri Ichikawa
Tomoko Nitta
Kaburagi Akiko
Yuzu Iijima
Reiko Sato
Donna Burke

T

㱗

t+1

T

Fig. 3. Stochastic dependency among F , Ψ and Λ.
2.3.1. Formulation
At each time step, t (t = 1, · · · , T ), we deﬁne an F0, spectrum andsound source as ft , ψt and λt , respectively. We also deﬁne F ={ft |t = 1, · · · , T }, Ψ = {ψt |t = 1, · · · , T } and Λ = {λt |t =1, · · · , T }. As a sound source, we consider a singing voice and theother sound, that is λt ∈ {sV , sN }.
We assume that stochastic dependency among Ft , Ψt , and Λt
can be expressed as show in Fig. 3. According to Fig. 3, we introduce the following conditional probability distributions: vocal/nonvocal probability p(λi |fi , ψi ), pitch likelihood p(ψi |fi ) and transition probability p(fi |fi−1 ). Pitch likelihood and sound source probability were explained in Sec. 2.1 and Sec. 2.2. We deﬁne transition
probability, p(fi |fi−1 ), asp(fi |fi−1 ) = G(fi ; fi−1 , WV ),

(8)

=
=

argmax log p(F |Ψ = O, Λ = (always sV ))F( TX
log p(λt = sV |ft , ψt = ot )argmax
FT

+

t=1

T
X

log p(ψt = ot |ft ) +

t=1

T
X

+β

T
X

log p(ψt = ot |ft ) +

t=1

T
X

(11)

t=1

We use Eq. (11) instead of Eq. (10), setting α = 0.2 and β = 0.8.
2.3.2. Viterbi Search
Since directly computing Eq. (11) is difﬁcult, we compute it recursively by using the following equations. We introduce an back pointer,
B(t, f ), and an accumulated probability, A(t, f ).
(1) Initialization(12)∀f A(1, f ) = α log p(λ1 |f, ψ1 ) + β log p(ψ1 |f )(2) Recursive calculation (t = 2, · · · , T )A(t, f )={A(t − 1, f  ) + α log p(λt |f, ψt )max
f

+β log p(ψt |f ) + log p(f |f  )}B(t, f )

=

(13)



argmax{A(t − 1, f ) + α log p(λt |f, ψt )f+β log p(ψt |f ) + log p(f |f  )}

(15)

fˆt

=

B(fˆt+1 )(16)

f(t = T − 1, · · · , 1)

3. EXPERIMENTS(10))
log p(ft |ft−1 )argmax A(T, f )We describe the construction of vocal/non-vocal models. As thetraining data, we used 21 songs of the 14 singers listed in Table 1,
which were taken from the “RWC Music Database: Popular” [11].
First, we computed boundaries between vocal and non-vocal sections by comparing polyphonic data and vocal-only data. The vocal
GMM was trained on feature vectors that were extracted from thevocal section of polyphonic data using the F0s estimated from vocalonly data. The non-vocal GMM was trained on feature vectors thatwere extracted from the non-vocal section of polyphonic data using
the F0s estimated from polyphonic data using PreFEst[1].

In practice, introduction of connection weights among vocal/nonvocal probability, pitch likelihood and transition probability are effective such as the following equation.
( TX
log p(λt = sV |ft , ψt = ot )F̂ = argmax α
t=1=

3.1. Construction of Vocal/Non-vocal Models

t=1

FT

fˆT

(9)

)
log p(ft |ft−1 )Piece Number
027
037
032, 078
049, 051
015, 041
056, 057
047
013
020
026
055
060
063
081, 091, 093, 097

We can obtain the sequence of F0s, F̂ = {fˆ1 , · · · , fˆT }, thatgives the highest probability by tracking back the back pointer.

where G(x; m, σ) represents the Gaussian distribution, and WV
denotes the changeability of F0. We set WV to 100 cent.
The purpose is, after observing the time series of spectra, O ={ot |t = 1, · · · , T }, and observing that sound source is singing
voice, sV , at every time step, to estimate a sequence of F0s F̂ thatmaximizes the following equation.
F̂

Gender
M
M
M
M
M
M
M
F
F
F
F
F
F
F

(14)

3.2. Conditions
We evaluated our method on 10 musical pieces taken from the “RWC
Music Database: Popular” [11] (Table 2). The singers of the musicalpieces used for training are not included in the 10 musical pieces
used for evaluation. To evaluate the effectiveness of our method, we
conducted experiments under the following six conditions:(i) Max density Select the F0s maximizing pitch density, without
considering vocal probability nor F0’s continuity.
(ii) PreFEst-back-end (baseline) Use the PreFEst-back-end [1],which tracks peak trajectories considering F0s’ continuity byintroducing a multiple-agent architecture.
(iii) MFCC Use our method with MFCCs as feature vectors.
(iv) MFCC+∆F0 Use our method with MFCCs and ∆F0 as feature
vectors.
(v) LPMCC Use our method with LPMCCs as feature vectors.
(vi) LPMCC+∆F0 (Proposed) Use our method with LPMCCs
and ∆F0 as feature vectors.
Estimation accuracies were evaluated by comparing the estimatedF0s with the correct F0s, obtained by estimating the F0s from vocalonly data. The evaluation was made during periods when a vocal
was present. We use two estimation accuracy indicators: pitch accuracy and chroma accuracy. The pitch accuracy is the probability ofV ­ 255

Table 2. Experimental results of F0 estimation for ﬁfteen musical pieces in RWC-MDB-P-2001.
Piece
number
No.007

Singer’s
genderFemaleNo.012

Male

No.019Male

No.021

FemaleNo.039Male

No.065

FemaleNo.075

FemaleNo.083

Male

No.088

Male

No.092

FemaleAverageAccuracy
indicator
Pitch acc.
Chroma acc.
Pitch acc.
Chroma acc.
Pitch acc.
Chroma acc.
Pitch acc.
Chroma acc.
Pitch acc.
Chroma acc.
Pitch acc.
Chroma acc.
Pitch acc.
Chroma acc.
Pitch acc.
Chroma acc.
Pitch acc.
Chroma acc.
Pitch acc.
Chroma acc.
Pitch acc.
Chroma acc.

Max density
76.5%
78.5%
74.0%
76.3%
59.8%
63.9%
80.1%
80.6%
78.0%
81.3%
73.8%
77.0%
79.9%
80.9%
72.5%
74.0%
67.5%
70.3%
76.6%
77.6%
73.9%
76.0%

PreFEst-back-end
87.3%
88.0%
78.4%
80.4%
62.5%
66.1%
83.1%
83.1%
81.1%
83.8%
79.3%
80.3%
83.7%
84.2%
72.0%
72.6%
76.4%
77.4%
77.2%
77.3%
78.1%
79.3%

correct pitch value. The chroma accuracy is the probability that thechroma (i.e. the note name) is correct. In other words, octave errors
are ignored.

3.3. Results and Discussions
Experimental results, listed in Table 2, show that our method improves the estimation accuracy from 78.1% to 84.3%. By this fact,
we can conﬁrm the effectiveness of our method. We can also ﬁndthat, introduction of ∆F0 improve the accuracy from 82.6% to 84.3%.
Though ∆F0 has not been used in conventional vocal/non-vocal discrimination method, we conﬁrmed that this feature is a good cue
for vocal/non-vocal discrimination. When we compare MFCCs and
LPMCCs, the accuracy for LPMCCs was 0.3% higher than that of
MFCCs.

4. CONCLUSION
We have described a method that estimates the F0s of vocal partin polyphonic audio signals. The basic ideas of our method are to
calculate the vocal probabilities of each F0 candidate and to track theF0 trajectory using these probability based Viterbi search. Thoughconventional vocal/non-vocal discrimination method cannot treat theharmonic structure of a particular F0, we made it possible by using
our accompaniment sound reduction. Experimental results showedthat our system accurately estimated the F0s of vocal and improve
estimation accuracy.
In the future, we plan to extend our method to the song in which
multiple singers sing together. We also plan to work on the detectionof vocal region (i.e. where the singer is singing in a musical piece).

Method
MFCC MFCC+∆F0
92.2%
92.9%
92.5%
93.2%
80.3%
81.8%
82.4%
83.9%
69.1%
70.0%
72.8%
73.8%
85.7%
86.5%
85.8%
86.5%
84.4%
86.3%
87.2%
89.2%
85.6%
88.9%
87.1%
89.0%
87.2%
91.0%
87.5%
91.3%
73.2%
76.4%
74.4%
77.7%
77.8%
84.9%
79.5%
85.8%
80.9%
81.3%
81.0%
81.4%
81.6%
84.0%
83.0%
85.2%

LPMCC
92.7%
93.0%
80.3%
82.4%
68.3%
72.4%
86.6%
86.7%
85.3%
88.1%
87.0%
87.4%
86.1%
86.6%
74.1%
75.4%
84.7%
85.7%
80.4%
80.5%
82.6%
83.8%

LPMCC+∆F0
93.1%
93.4%
82.3%
84.5%
70.2%
74.2%
87.5%
87.5%
86.2%
89.1%
90.2%
90.3%
90.4%
90.7%
76.9%
78.3%
84.6%
85.5%
81.3%
81.3%
84.3%
85.5%

5. REFERENCES[1] M. Goto,“A real-time music-scene-description system:
predominant-F0 estimation for detecting melody and bass lines
in real-world audio signals,” Speech Communication, vol. 43,
no. 4, pp. 311–329, 2004.
[2] M. Marolt, “Gaussian mixture models for extraction of melodic
lines from audio recordings,” in Proc. ISMIR, pp.80–83, 2004[3] J. Eggink and G. J. Brown, “Extracting melody lines from complex audio,” in Proc. ISMIR, pp.84–91, 2004.
[4] M. P. Ryynanen and A. Klapuri, “Note event modeling for audio
melody extraction,” in Online Proc. MIREX2005, 2005.
[5] A. L. Berenzweig and D. P. W. Ellis, “Locating singing voice
segments within music signals,” in Proc. WASPAA, 2001.
[6] W. Tsai and H. Wang, “Automatic detection and tracking of target singer in multi-singer music recordings,” in Proc. ICASSP,
pp. 221–224, 2004.
[7] T. L. Nwe, “Automatic detection of vocal segments in popularsongs,” in Proc. ISMIR, pp. 138–145, 2004.
[8] H. Fujihara, T. Kitahara, M. Goto, K. Komatani, T. Ogata, and
H. G. Okuno, “Singer identiﬁcation based on accompanimentsound reduction and reliable frame selection,” in Proc. ISMIR,
pp. 329–336, 2005.
[9] J. A. Moorer, “Signal processing aspects of computer music: A
survey,” Proceedings of the IEEE, vol. 65, no. 8, pp. 1108–1137,
1977.
[10] Y. Ohishi, M. Goto, K. Itou, and K. Takeda, “Discrimination
between singing and speaking voices,” in Proc. Eurospeech, pp.
1141–1144, 2005.
[11] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka, “RWC
Music Database: Popular, classical, and jazz music databases,”in Proc. ISMIR, pp. 287–288, 2002.

V ­ 256