MUSICAL NOTE ESTIMATION FOR F0 TRAJECTORIES OF SINGINGVOICES BASED ON A BAYESIAN SEMI-BEAT-SYNCHRONOUS HMM
Ryo Nishikimi1

1

Eita Nakamura1
Katsutoshi Itoyama1
Kazuyoshi Yoshii1
Graduate School of Informatics, Kyoto University , Japan{nishikimi, enakamura}@sap.ist.i.kyoto-u.ac.jp, {itoyama, yoshii}@kuis.kyoto-u.ac.jp

ABSTRACTThis paper presents a statistical method that estimates a sequence of discrete musical notes from a temporal trajectoryof vocal F0s. Since considerable effort has been devoted toestimate the frame-level F0s of singing voices from music
audio signals, we tackle musical note estimation for thoseF0s to obtain a symbolic musical score. A naı̈ve approach
to musical note estimation is to quantize the vocal F0s at
a semitone level in every time unit (e.g., half beat). Thisapproach, however, fails when the vocal F0s are significantly deviated from those specified by a musical score.
The onsets of musical notes are often delayed or advanced
from beat times and the vocal F0s fluctuate according tosinging expressions. To deal with these deviations, we propose a Bayesian hidden Markov model that allows musicalnotes to change in semi-synchronization with beat times.
Both the semitone-level F0s and onset deviations of musical notes are regarded as latent variables and the frequency
deviations are modeled by an emission distribution. The
musical notes and their onset and frequency deviations arejointly estimated by using Gibbs sampling. Experimental results showed that the proposed method improved theaccuracy of musical note estimation against baseline methods.
1. INTRODUCTION
Singing voice analysis is one of the most important topics
in the field of music information retrieval because singing
voice usually forms the melody line of popular music andit has a strong impact on the mood and impression of amusical piece. The widely studied tasks in singing voice
analysis are fundamental frequency (F0) estimation [1, 4,
5, 8, 10, 15, 22] and singing voice separation [9, 13] for music audio signals. These techniques can be used for singer
identification [11, 23], Karaoke systems based on singing
voice suppression [2,20], and a music listening system thathelps a user focus on a particular musical element (e.g., vocal part) for deeper music understanding [7].
c Ryo Nishikimi, Eita Nakamura, Katsutoshi Itoyama,⃝
Kazuyoshi Yoshii. Licensed under a Creative Commons Attribution 4.0
International License (CC BY 4.0). Attribution: Ryo Nishikimi, Eita
Nakamura, Katsutoshi Itoyama, Kazuyoshi Yoshii. “Musical Note Estimation for F0 Trajectories of Singing Voices based on a Bayesian Semibeat-synchronous HMM”, 17th International Society for Music Information Retrieval Conference, 2016.

Pitch

Musical Score Component

Pitch+ Onset Deviation

Time

Pitch

+ Frequency Deviation

Time

Time
Figure 1: The process generating F0 trajectories of singing
voices.

In this study we tackle a problem called musical note estimation that aims to recover a sequence of musical notesfrom an F0 trajectory of singing voices. While a lot ofeffort has been devoted to F0 estimation of singing voices,musical note estimation should be investigated additionallyto complete automatic music transcription, i.e., convert theestimated F0 trajectory to a musical score containing only
discrete symbols. If beat information is available, a naı̈ve
approach to this problem is to quantize the vocal F0s contained in every time unit (e.g., half beat) into a semitonelevel F0 with a majority vote [7]. This approach, however,often fails to work when the vocal F0s are significantly
deviated from exact semitone-level F0s specified by a musical score or the melody is sung in a tight or lazy singing
style such that the onsets of musical notes are significantly
advanced or delayed from exact beat times.
To solve this problem, we propose a statistical method
based on a hidden Markov model (HMM) that represents
how a vocal F0 trajectory is generated from a sequence oflatent musical notes (Fig. 1). The F0s of musical notes ina musical score can take only discrete values with the interval of semitones and tend to vary at a beat, half-beat,or quarter-beat level. The vocal F0 trajectory in an actual
performance, on the other hand, is a continuous signal thatcan dynamically and smoothly vary over time. To deal withboth types of F0s from a generative viewpoint, we formulate a semi-beat-synchronous HMM (SBS-HMM) allowing the continuous F0s of a sung melody to deviate fromthe discrete F0s of written musical notes along the time andfrequency directions. In the proposed HMM, the semitonelevel F0s and onset deviations of musical notes are encoded

461

462

Proceedings of the 17th ISMIR Conference, New York City, USA, August 7-11, 2016as latent variables and the F0 deviations of musical notes
are modeled by emission probability distributions. Givenan F0 trajectory and beat times, all the variables and distributions are estimated jointly using Gibbs sampling.

Music Audio Signal

Vocal F0 Estimation

Beat Tracking

2. RELATED WORKThis section introduces related work on singing voices.

Note Estimation

2.1 Pitch Estimation of Singing Voice
Many studies on estimating a vocal F0 trajectory in a music
audio signal have been conducted [1,4,5,8,10,15,22]. Subharmonic summation (SHS) [8] is a method in which thefundamental frequency for each time is determined by calculating the sum of the powers of the harmonic components of each candidate fundamental frequency {f0 , . . . , fM }.
PreFEst [5] is a method that estimates the F0 trajectories ofa melody and a bass line by extracting the most predominant harmonic structure from a polyphonic music audio
signal. Ikemiya et al. [10] proposed a method in which singing voice separation and F0 estimation are performed mutually. First a singing voice is separated, from the spectrogram obtained by the short-time Fourier transform (STFT)
for a music audio signal, by using a robust principal component analysis (RPCA) [9], and then a vocal F0 trajectory
is obtained with the Viterbi algorithm by using SHS fora separated singing voice. Salamon et al. [22] used thecharacteristics of vocal F0 contours for melody extraction.
Durrieu et al. [4] proposed a method for melody extraction in which the main melody is represented as a sourcefilter model and the accompaniment of the music mixture isrepresented as a non-negative matrix factorization (NMF)based model. De Cheveigné et al. [1] proposed a autocorrelation based method for fundamental frequency estimation which is expanded to decrease an error rate. Thismethod is called YIN. Mauch et al. [15] extended YIN in aprobabilistic way to output multiple pitch candidates. Thismethod is called pYIN.
2.2 Note Estimation of Singing Voice
A method for estimating the sequence of musical notes byquantizing pitches of a vocal F0 trajectory has been proposed. A majority-vote method described in Sec. 1 was
implemented in Songle [7]. The method has a limit because it doesn’t consider the singing expression nor the
typical occurrence of pitches in succession. Paiva et al.
[17] proposed a method that has five stages and detects
melody notes in polyphonic musical signals, and Raphael[19] proposed an HMM-based method that simultaneously
estimates rhythms, tempos, and notes from a solo singing
voice acoustic signal. Poliner et al. [18] proposed a method
based on a support vector machines (SVM) classifier which
doesn’t need the assumption that a musical pitch is realized
as a set of harmonics of a particular fundamental. Laaksonen [12] proposed a melody transcription method that uses
chord information, and Ryynänen et al. [21] proposed amethod for transcribing the melody, bass line, and chords
in polyphonic music. A software tool called Tony devel-

Figure 2: Overview of the proposed musical note estimation method based on a semi-beat-synchronous HMM.
oped by Mauch et al. [14] estimates musical notes fromthe output of pYIN by Viterbi-decoding of an HMM.
2.3 Analysis of Vocal F0 Trajectories
Studies on extracting the personality and habit of singing
expression from vocal F0 trajectories have been conducted.
Ohishi et al. [16] proposed a model that represents the generating process of vocal F0 trajectories in consideration of
the time and frequency deviations. In that model the vocal
F0 trajectory consists of three components: note, expression, and fine deviation components. The note component
contains the note transition and overshoot, and the expression component contains vibrato and portamento. The noteand expression components are represented as the outputsof second-order linear systems driven by the note and expression commands. The note and expression commands
represent the sequence of musical notes and the musicalexpressive intentions, respectively. The note command andthe expression command are represented with HMMs. Although the method can extract the personality of the singing
expression from vocal F0 trajectories, it assumes that themusic score is given in advance and cannot be directly applied for note estimation.
3. PROPOSED METHODThis section explains the proposed method for estimating
a sequence of latent musical notes from the observed vocal
F0 trajectories by formulating an SBS-HMM which represents the generating process of the observations. An observed F0 trajectory is stochastically generated by imparting frequency and onset deviations to a step-function-like
F0 trajectory that varies exactly on a 16th-note-level grid
according to a music score. The semitone-level F0s (called
pitches for simplicity in this paper) between adjacent gridlines and the onset deviations are represented as latent variables (states) of the HMM. Since the frequency deviations
are represented by emission probability distributions of theHMM, a semi-beat-synchronous step-function-like F0 trajectory is generated in the latent space and its finely-fluctuated version is then generated in the observed space.
3.1 Problem Specification
The problem of musical note estimation is formally defined(Fig. 2) as follows:Proceedings of the 17th ISMIR Conference, New York City, USA, August 7-11, 2016

T

Input: a vocal F0 trajectory X = {xt }t=1 and 16th-noteN
level beat times ψ = {ψn }n=1 automatically estimatedfrom a music audio signal.
N
Output: a sequence of pitches Z = {zn }n=1 .

Here, t is the time frame index, T is the number of timeframes in the target signal, xt indicates a log-frequency incents at frame t, N is the number of beat times, ψn is then-th beat time, and zn indicates a pitch between ψn−1 andψn taking one of {µ1 , . . . , µK }, where K is the numberof kinds of pitches that can appear in a music score. The
beginning and end of music are represented as ψ0 = 1 andψN = T +1 respectively. In this paper we assume thateach zn corresponds to a 16th note for simplicity. A longerNmusical note is represented by a subsequence of {zn }n=1having the same pitch in succession.
3.2 Model Formulation
We explain how to formulate the SBS-HMM that simultaneously represents the pitch transitions and onset and frequency deviations of musical notes.
A sequence of latent pitches Z forms a first-order Markov
chain given by(1)

where A = [aT1 , · · · , aTK ] is a K-by-K transition proba∑Kbility matrix such that k=1 ajk = 1 for any j. The initiallatent state z1 is determined as follows:

z1 |π ∼ Categorical(z1 |π),(2)

where π = [π1 , · · · , πK ] is a K-dimensional vector such∑K
that k=1 πk = 1.
T

3.2.2 Modeling Onset Deviations

The onset deviations of musical notes τ = {τn }Nn=1 arerepresented as discrete latent variables taking integer values between −G and G. Let ϕn = ψn + τn be the actual
onset time of the n-th musical note. Note that τ0 = 0 andτN = 0 at the beginning and end of a vocal F0 trajectory.
We assume that τn is stochastically generated as follows:τn |ρ ∼ Categorical(τn |ρ),where p(xt |zn = k) is the emission probability of eachframe. To balance the effects of transition probabilities
and emission probabilities, we exponentiate the product ofemission probabilities of frames in a beat interval by the
number of frames in a beat interval. We use as p(xt |zn )the Cauchy distribution, which is robust against outliers
and is defined by
Cauchy(x; µ, λ) =λ,π {(x − µ)2 + λ2 }

(3)

(5)

where µ is a location parameter that defines the mode ofthe distribution and λ is a scale parameter. When the pitchof the n-th beat interval is zn = k, µ takes the value µk .
The scale parameter takes a value that does not depend on
the pitch zn .
Since actual vocal F0s are significantly deviated fromthose specified by a musical score, the scale parameter of a
Cauchy distribution is allowed to change according to thedifference of adjacent F0s; i.e., ∆xt ≡ xt − xt−1 . The
scale parameter is set to be proportional to the absolutevalue of ∆xt and defined for each frame t as follows:λt = c · |∆xt | + d,

3.2.1 Modeling Pitch Transitions

zn |zn−1 , A ∼ Categorical(zn |azn−1 ),

463

(6)

where c is an coefficient and d is a constant term. If d = 0,p(xt |zn ) cannot be calculated when λt = 0 and ∆xt = 0.
To avoid this problem, we introduce the constant term d.
3.2.4 Incorporating Prior Distributions
We put conjugate Dirichlet priors on model parameters A,
π, and ρ as follows:
aj ∼ Dirichlet(aj |ξj ),(7)

ρ ∼ Dirichlet(ρ|η),(9)

π ∼ Dirichlet(π|ζ),(8)where ξj = [ξ1 , . . . , ξK ]T and ζ = [ζ1 , . . . , ζK ]T are Kdimensional vectors and η = [η−G , . . . , ηG ]T is a (2G +1)-dimensional vector.
We then put on gamma priors on nonnegative Cauchy
parameters c and d as follows:c ∼ Gamma(c|c0 , c1 ),d ∼ Gamma(d|d0 , d1 ),

(10)
(11)

where ρ = [ρ−G , . . . , ρG ]T is a (2G+1)-dimensional vec∑G
tor such that g=−G ρg = 1.

where c0 and d0 are shape parameters and c1 and d1 arerate parameters.

3.2.3 Modeling Frequency Deviations

3.3 Bayesian Inference

The observed F0 xt (ϕn−1 ≤ t < ϕn ) is stochasticallygenerated by imparting a probabilistic frequency deviation
to the semitone-level pitch µzk assigned to each beat interval. Assuming that xt is independently generated at eachframe, the emission probability of the n-th beat interval in
the case of zn = k, τn−1 = f, τn = g is given by ϕ −ϕ1n −1
 ϕ∏
 n n−1bnkf g ≡p(xt |zn = k)
,
(4)



The goal of Bayesian inference is to calculate the posterior
distribution p(Z, τ , A, π, ρ, c, d|X). Since this computation is analytically intractable, we use Markov chain Monte
Carlo (MCMC) methods for sampling the values of thosevariables. Let Θ = {A, π, ρ} be a set of model parameters. To get samples of Θ, the Gibbs sampling algorithm isused. To get samples of sequential latent variables Z andτ , on the other hand, a kind of blocked Gibbs sampling algorithms called a forward filtering-backward sampling algorithm is used. These steps are iterated in a similar way

t=ϕn−1

464

Proceedings of the 17th ISMIR Conference, New York City, USA, August 7-11, 2016
to an expectation maximization (EM) algorithm called the
Baum-Welch algorithm used for unsupervised learning ofan HMM. Since the conjugacy is not satisfied for the distributions regarding c and d, we use the Metropolis-Hastings(MH) algorithm.
3.3.1 Inferring Latent Variables Z and τ
We explain how to sample a sequence of latent variables Zand τ . For each beat interval, we calculate the probability
given by

3.3.2 Learning Model Parameters A, π, and ρ
We explain how to learn the values of Θ. In a sequence ofNlatent variables {zn , τn }n=1 which are sampled in backward sampling, the number of transitions such that zn = j
and zn+1 = k is represented as sjk and the number of onset deviations such that τn = g is represented as ug . The
value of vk is 1 at z1 = k, and else where is 0. The parameters ajk , ρg and πk are updated by sampling from theposterior distributions given byp(aj |ξj + sj ) = Dirichlet(aj |ξ j + sj ),βnjf = p(zn = j, τn = f |zn+1:N , τn+1:N , x1:T ), (12)where zn+1:N , τn+1:N , and x1:T represent zn+1 , . . . , zN ,
τn+1 , . . . , τN , and x1 , . . . , xT , respectively. The latent variables of the n-th beat interval (zn , τn ) are sampled in accordance with βnjf . The calculation of Eq. (12) and thesampling of the latent variables are performed by using forward filtering-backward sampling.
In forward filtering, we recursively calculate the probability given by
αnkg = p(X10τ1 , . . . , X(n−1)τn−2 f , Xnf g , zn =k, τn =g),where Xnf g represents the observations xt in the beat interval from ϕn−1 to ϕn when τn−1 = f and τn = g. αnkg
is calculated as follows:
α1kg = p(X10g , z1 = k, τ1 = g)
= p(X10g |z1 = k, τ1 = g)p(z1 = k)p(τ1 = g)
= b1k0g πk ρg ,(13)

αnkg = p(X10τ1 , . . . , Xnf g , zn = k, τn = g)
=

G
∑

f =−G

·

K
∑

p(Xnf g |zn = k, τn−1 = f, τn = g)p(X10τ1 , . . . , X(n−1)τn−2 f , zn−1 =j, τn−1 =f )· p(zn =k|zn−1 =j)p(τn =g)
=

f =−G

bnkf gK∑α(n−1)jf ajk ρg .

(19)

To estimate the parameters c and d, we use the MH algorithm. It is hard to analytically calculate the posterior distributions of c and d because a Cauchy distribution doesn’t
have conjugate prior distributions. When the values of cand d are respectively ci and di , we define proposal distributions of c and d as follows:qc (c|ci ) = Gamma(c|γci , γ),(20)

qd (d|di ) = Gamma(d|δdi , δ),(21)

where γ and δ are hyperparameters of the proposal distributions. Using the value of c∗ sampled from qc (c|ci ), we
calculate the acceptance ratio given by{}
fc (c∗ )qc (ci |c∗ )∗gc (c , ci ) = min 1,,(22)
fc (ci )qc (c∗ |ci )fc (c) ≡ p(c|x1:T , z1:N , τ1:N , Θ, di )=βnjf ∝ p(X(n+1)f g |zn+1 = k, τn = f, τn+1 = g)
· p(zn+1 = k|zn = j)p(τn+1 = g)

(15)

Specifically, the latent variables (zN , τN ) are sampled inaccordance with the probability given byβN jf ∝ αN jf .

p(π|ζ + v) = Dirichlet(π|ζ + v),

3.3.3 Learning Cauchy Parameters c and d(14)

In backward sampling, Eq. (12) is calculated in the nth beat interval by using the value of αnkg , and the states(zn , τn ) are sampled recursively. When the (n+1)-th sampled states are (zn+1 , τn+1 ) = (k, g), βnjf is calculated asfollows:= b(n+1)kf g ajk ρg αnjf .

(18)

where sj = [sj1 , . . . , sjK ]T , u = [u−G , . . . , uG ]T , andv = [v1 , . . . , vK ]T .

∝j=1

· p(X10τ1 , . . . , Xnτn f , zn = j, τn = f )p(ρ|η + u) = Dirichlet(ρ|η + u),

where fc (c) is a likelihood function given by

j=1

G
∑

(17)

(16)

N
∏

ρn bnzn τn−1 τn

N
∏

ρn

·

azn−1 zn πz1 Gamma(c|c0 , c1 ),

n=1

n=1
N
∏


n −1
 ϕ∏


t=ϕn−1N∏

azn−1 zn πz1 q(c)n=2Cauchy(xt |µzn , λct ) ϕ −ϕ1
 n n−1


(23)

n=2

λct = ci · ∆xt + di ,(24)

Finally, if the value of gc (c∗ , ci ) is larger than the random
number r sampled from a uniform distribution on the interval [0, 1], then ci+1 = c∗ , and otherwise ci+1 = ci , wherec0 is sampled from the prior distribution q(c).
The value of d is updated in the same way as that of c.
Using the value of d∗ sampled from qd (d|di ), we calculate
the acceptance criteria given by{}fd (d∗ )qd (di |d∗ )∗gd (d , di ) = min 1,,(25)
fd (di )qd (d∗ |di )Proceedings of the 17th ISMIR Conference, New York City, USA, August 7-11, 2016465

where fd (d) is a likelihood function given by

Model

Concordance rate

fd (d) ≡ p(d|x1:T , z1:N , τ1:N , Θ, ci+1 )SBS-HMM
Majority vote
Frame-based HMM
BS-HMM

66.3 ± 1.0
56.9 ± 1.1
56.1 ± 1.1
67.0 ± 1.0=N∏ρn bnzn τn−1 τn

n=1


n −1
 ϕ∏

N
∏

azn−1 zn πz1 q(c)n=2

 ϕ −ϕ1
 n n−1

N
∏

ρn

·

azn−1 zn πz1 Gamma(d|d0 , d1 ),

n=1
N
∏



t=ϕn−1

Cauchy(xt |µzn , λdt )



(26)

n=2

λdt = ci+1 · ∆xt + di ,(27)

Finally, if the value of gd (d∗ , di ) is larger than the random
number r sampled from a uniform distribution on the interval [0, 1], then di+1 = d∗ , and otherwise di+1 = di , whered0 is sampled from the prior distribution q(d).
3.4 Viterbi Decoding
A latent sequence of musical notes is estimated by using
the Viterbi algorithm that uses the parameters at the time
when the likelihood given byp(x1:T ) =

K ∑
G
∑

αnkg

(28)

90
80
70
60
50
40
30

Majority vote Frame-based
HMM

BS-HMM

SBS-HMM(proposed)

Figure 3: Concordance rates [%]. In the box plots, the redline indicates the median, the blue box indicates the range
from the first to third quantile, the black cross indicates themean, and the outliers are plotted with red crosses.
4. EVALUATION
We conducted an experiment to evaluate the proposed andprevious methods in the accuracy of estimating musicalnotes from vocal F0 trajectories.
4.1 Experimental Conditions

k=1 g=−Gis the maximum in the learning process. The musical notesthat we want to estimate are the latent variables that maximize the value given by p(Z, τ |X). In the Viterbi algorithm, we define ωnkg as follows:ωnkg =
max lnp(X10τ1 , . . . , Xnτn−1 g , z1:n−1 , zn =k, τ1:n−1 , τn =g),z1:n−1τ1:n−1(29)
and ωnkg is calculated recursively with the equations
ω1kg = ln ρg + ln b1k0g + ln πk ,
(30)
[
]
{
}
ωnkg = ln ρg + max ln bnkf g + max ln ajk +ω(n−1)jf .
jfTable 1: Average concordance rates and their standard errors.
Concordance rate [%]

∝

(31)

In the recursive calculation of ωnkg , when the states thatmaximize the value of ωnkg are zn−1 = j, τn−1 = f ,(z)(τ )those states are memorized as hnk = j, hng = f . AfK,G
ter calculating {ωN kg }k=1,g=−G with Eqs. (30) and (31),N

the sequence of latent variables {zn , τn }n=1 is recursively
estimated with the equations given by(zN , τN ) = arg max{ωnkg },(32)

zn = h(n+1)zn+1 ,

(z)

(33)

(τ )
τn = h(n+1)τn+1 .

(34)

k,g

The note sequence is retrieved by revising the onset deviaNtions represented by the estimated latent variables {τn }n=1 .

The 100 pieces of popular music in RWC database [6] wereused for the experiments. For each song, we trained model
parameters, estimated the sequence of musical notes, andmeasured the accuracy of estimated musical notes. The
input F0 trajectories were obtained from monaural music
acoustic signals by the method of Ikemiya et al. [10]. We
used the beat times obtained by a beat tracking system by
Durand et al. [3]. This system estimates the beat times inunits of a whole note, and the interval between adjacentbeat times were divided into 16 equal intervals to obtain
the beat times for 16th-note units were calculated.
For the proposed method, the sequence of musical notes
was estimated with the Viterbi algorithm. The hyperparameters were ξ = 1, ζ = 1, η = 1, c0 = d1 = d0 =d1 = 1, where 1 and 1 respectively represent the matrix
and vector whose elements are all ones. The parameters of
the proposal distributions were γ = δ = 1. The maximum
value G that τn could take was G = 5 (i.e., 50 cents).
A majority-vote method was tested as a baseline. It
estimates a musical note in each time unit corresponding
to a 16th note by taking a majority vote for vocal F0s in
the time unit. For comparison, a frame-based HMM anda beat-synchronous HMM (BS-HMM) were also tested.
The frame-based HMM assumes that all beat intervals have
only on frame. The BS-HMM is the same as SBS-HMM
except that the onset deviation is not considered.
The estimated sequence of musical notes was comparedwith the ground-truth MIDI data synchronized to the vocal
melody, and the concordance rate, i.e., the rate of frames
in which pitches are correctly estimated, was used as theevaluation measure.

Proceedings of the 17th ISMIR Conference, New York City, USA, August 7-11, 2016
0.6
0.5
0.4
0.3
0.2
0.1
0.0

4000
Pitch [cent]

Probability

466

−4

−2
0
2
Onset deviation [time frame]

4

3800
3600
3400
3200
3000
0

700
600
500
400
300
200
100
0
−15

50

100

150

200
250
Time [ms]

300

350

400

450

(a) Onset deviation
5100
Pitch [cent]

The number of frames

Figure 4: Example of a learned distribution of the model
parameter ρ.

−10
−5
0
5
(Estimated pitch) - (Correct pitch) [semitone]

10

Figure 5: A example of pitch estimation error. The casethat an estimated pitch is equal to a correct pitch is omitted.
4.2 Experimental Results
The results of note estimation are listed in Table 1 andFigure 3. The proposed model clearly outperformed the
majority-vote method and the frame-based HMM in the
average concordance rate. On the other hand, the averageconcordance rates for the proposed model and BS-HMM
were similar and the difference was not statistically significant.
The results indicate that the model of music scores bytransition probabilities and that of frequency deviations by
output probabilities are both effective for improving theaccuracy of musical note estimation. The cause of the result that the model of onset deviations did not improve theaccuracy is probably that the model parameter ρ was notproperly learned (Fig. 4). Capturing onset deviations by asingle categorical distribution would be difficult, since onset depends on the duration of the pitches on either side of
the onset, and on the over all tempo of the song. It would
be necessary to model onset deviations in detail, for example by using a separate hidden state to represent the F0s
during pitch transition.
4.2.1 Pitch Estimation Error
Two types of errors were mainly observed (Fig. 5). The
first type was caused by singing styles of singers, and appears as errors that span one semitone and two semitones.
This means that the frequency deviations affect the accuracy of note estimation. The second type was caused by theerror of F0 estimation, and appears as the errors that spanseven semitones and twelve semitones. The intervals ofseven semitones and 12 semitones correspond to a perfectfifth and an octave, respectively.
4.2.2 Singing Style Extraction and Robustness
Example results of note estimation in Figure 6 show thepotential of the proposed model to capture the singers’
singing style. In the upper figure, the onset at the first beatis significantly delayed from the beat time. Whereas the
proposed model correctly detected the delayed onset, themajority-vote method mis-identified the beat position of

5050
5000
4950
4900

0

50 100 150 200 250 300 350 400 450 500 550 600 650 700
Time [ms](b) Frequency deviation

Figure 6: Examples of note estimation results. The pink,
blue, green, red, and black vertical lines respectively indicate a MIDI note which is the ground-truth, the F0 trajectory of a singing voice including onset deviations, thepitches estimated by the majority-vote method, the pitches
estimated by the proposed method, and the beat times estimated in advance.
the pitch onset. The lower figure is an example of vibrato.
With the majority-vote method, the estimation result wasaffected by the large frequency deviation. With the proposed method, on the other hand, the robustness due to the
Cauchy distribution enabled the correct estimation of thepitch without being affected by the vibrato.

5. CONCLUSION
This paper presented a method for estimating the musicalnotes of music from the vocal F0 trajectory. When modeling the process generating the vocal F0 trajectory, we
considered not only the musical score component but also
onset deviation and frequency deviation. The SBS-HMM
estimated pitches more accurately than the majority-votemethod and the frame-based method.
The onset deviation and frequency deviation that wereobtained using the proposed method are important for grasping the characteristics of singing expression. Future work
includes precise modeling of vocal F0 trajectories basedon second-order filters and extraction of individual singing
expression styles. In the proposed method, F0 estimation,
beat tracking, and musical note estimation are conducted
separately. It is necessary to integrate these methods. The
proposed method cannot deal with the non-vocal regionsin actual music, so we plan to also appropriately deal with
non-vocal regions.
Acknowledgement: This study was partially supported by JST
OngaCREST Project, JSPS KAKENHI 24220006, 26700020,
26280089, 16H01744, and 15K16054, and Kayamori Foundation.

Proceedings of the 17th ISMIR Conference, New York City, USA, August 7-11, 2016
6. REFERENCES[1] A. de Cheveigné and H. Kawahara. YIN, a fundamental frequency estimator for speech and music. The Journal of the Acoustical Society of America, 111(4):1917–
1930, 2002.
[2] A. Dobashi, Y. Ikemiya, K. Itoyama, and K. Yoshii. A
music performance assistance system based on vocal,
harmonic, and percussive source separation and content visualization for music audio signals. SMC, 2015.

467[12] A. Laaksonen. Automatic melody transcription basedon chord transcription. In The International Society forMusic Information Retrieval (ISMIR), pages 119–124,
2014.
[13] Y. Li and D. Wang. Separation of singing voice frommusic accompaniment for monaural recordings. IEEE
Transactions on Audio, Speech, and Language Processing, 15(4):1475–1487, 2007.

[3] S. Durand, J. P. Bello, B. David, and G. Richard.
Downbeat tracking with multiple features and deepneural networks. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), pages 409–413. IEEE, 2015.

[14] M. Mauch, C. Cannam, R. Bittner, G. Fazekas, J Salamon, J. Dai, J. Bello, and S Dixon. Computer-aided
melody note transcription using the Tony software: Accuracy and efficiency. In Proceedings of the First International Conference on Technologies for Music Notation and Representation - TENOR2015, pages 23–30,
Paris, France, 2015.

[4] J.-L. Durrieu, G. Richard, B. David, and C. Févotte.
Source/filter model for unsupervised main melody extraction from polyphonic audio signals. IEEE Transactions on Audio, Speech, and Language Processing,,
18(3):564–575, 2010.

[15] M. Mauch and S. Dixon. pYIN: A fundamental frequency estimator using probabilistic threshold distributions. In 2014 IEEE International Conference on
Acoustics, Speech and Signal Processing (ICASSP),
pages 659–663. IEEE, 2014.

[5] M. Goto. PreFEst: A predominant-F0 estimation
method for polyphonic musical audio signals. Proceedings of the 2nd Music Information Retrieval Evaluation
eXchange, 2005.

[16] Y. Ohishi, H. Kameoka, D. Mochihashi, and
K. Kashino. A stochastic model of singing voice F0
contours for characterizing expressive dynamic components. In INTERSPEECH, pages 474–477, 2012.

[6] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka.
RWC music database: Popular, classical and jazz music databases. In The International Society for Music
Information Retrieval (ISMIR), volume 2, pages 287–
288, 2002.

[17] R. P. Paiva, T. Mendes, and A. Cardoso. On the detection of melody notes in polyphonic audio. In The
International Society for Music Information Retrieval(ISMIR), pages 175–182, 2005.

[7] M. Goto, K. Yoshii, H. Fujihara, M. Mauch, andT Nakano. Songle: A web service for active music listening improved by user contributions. In The International Society for Music Information Retrieval (ISMIR), pages 311–316, 2011.
[8] D. J. Hermes. Measurement of pitch by subharmonicsummation. The journal of the acoustical society ofAmerica, 83(1):257–264, 1988.
[9] P.-S. Huang, S. D. Chen, P. Smaragdis, and
M. Hasegawa-Johnson. Singing-voice separation frommonaural recordings using robust principal component
analysis. In 2012 IEEE International Conference on
Acoustics, Speech and Signal Processing (ICASSP),
pages 57–60. IEEE, 2012.
[10] Y. Ikemiya, K. Yoshii, and K. Itoyama. Singing voice
analysis and editing based on mutually dependent F0
estimation and source separation. In 2015 IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), pages 574–578. IEEE, 2015.
[11] Y. E. Kim and B. Whitman. Singer identification in
popular music recordings using voice coding features.
In Proceedings of the 3rd International Conference
on Music Information Retrieval, volume 13, page 17,
2002.

[18] G. E. Poliner and D. P. W. Ellis. A classification approach to melody transcription. The International Society for Music Information Retrieval (ISMIR), pages
161–166, 2005.
[19] C. Raphael. A graphical model for recognizing sung
melodies. In The International Society for Music Information Retrieval (ISMIR), pages 658–663, 2005.
[20] M. Ryynänen, T. Virtanen, J. Paulus, and A. Klapuri. Accompaniment separation and karaoke application based on automatic melody transcription. In
2008 IEEE International Conference on Multimedia
and Expo, pages 1417–1420. IEEE, 2008.
[21] M. P. Ryynänen and A. P. Klapuri. Automatic transcription of melody, bass line, and chords in polyphonic music. Computer Music Journal, 32(3):72–86,
2008.
[22] J. Salamon and E. Gómez. Melody extraction frompolyphonic music signals using pitch contour characteristics. IEEE Transactions on Audio, Speech, andLanguage Processing, 20(6):1759–1770, 2012.
[23] W.-H. Tsai and H.-M. Wang. Automatic singer recognition of popular music recordings via estimation andmodeling of solo vocal signals. IEEE Transactions on
Audio, Speech, and Language Processing, 14(1):330–
341, 2006.