SCALE- AND RHYTHM-AWARE MUSICAL NOTE ESTIMATION FOR
VOCAL F0 TRAJECTORIES BASED ON A SEMI-TATUM-SYNCHRONOUS
HIERARCHICAL HIDDEN SEMI-MARKOV MODEL
Ryo Nishikimi1 Eita Nakamura1 Masataka Goto2 Katsutoshi Itoyama1 Kazuyoshi Yoshii1,3
1Graduate School of Informatics, Kyoto University, Japan 3 RIKEN AIP, Japan
2
National Institute of Advanced Industrial Science and Technology (AIST), Japan{nishikimi, enakamura, itoyama, yoshii}@sap.ist.i.kyoto-u.ac.jp, m.goto@aist.go.jp

ABSTRACTThis paper presents a statistical method that estimates a sequence of musical notes from a vocal F0 trajectory. Since
the onset times and F0s of sung notes are considerably deviated from the discrete tatums and pitches indicated in amusical score, a score model is crucial for improving timefrequency quantization of the F0s. We thus propose a hierarchical hidden semi-Markov model (HHSMM) that combines a score model representing the rhythms and pitchesof musical notes with musical scales with an F0 model representing time-frequency deviations from a note sequencespecified by a score. In the score model, musical scales
are generated stochastically. Note pitches are then generated according to the scales and note onsets are generated
according to a Markov process defined on the tatum grid.
In the F0 model, onset deviations, smooth note-to-note F0transitions, and F0 deviations are generated stochasticallyand added to the note sequence. Given an F0 trajectory,
our method estimates the most likely sequence of musicalnotes while giving more importance on the score model
than the F0 model. Experimental results showed that the
proposed method outperformed an HMM-based method
having no models of scales and rhythms.
1. INTRODUCTION
Singing voice analysis is important for music information
retrieval because a singing voice usually forms a large partof the melody line of popular music, and provides much information about music. Singing voice analysis techniquessuch as vocal F0 estimation [1,3,7,9,14] and singing voice
separation [8, 12] have actively been studied and applied
to singer identification [10, 22], karaoke generation [19],query-by-humming [8], and active music listening [6]. To
leverage musical information conveyed by singing voices,
it is helpful to convert a vocal F0 trajectory to a musicalscore containing only discrete symbols.
c Ryo Nishikimi, Eita Nakamura, Masataka Goto, Kat⃝
sutoshi Itoyama, Kazuyoshi Yoshii. Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Attribution:
Ryo Nishikimi, Eita Nakamura, Masataka Goto, Katsutoshi Itoyama,
Kazuyoshi Yoshii. “Scale- and Rhythm-Aware Musical Note Estimation
for Vocal F0 Trajectories Based on a Semi-Tatum-Synchronous Hierarchical Hidden Semi-Markov Model”, 18th International Society for Music Information Retrieval Conference, Suzhou, China, 2017.

6FDOH

%DU
&PDMRU'HSHQGRQWKHVFDOH%DU&PDMRU%DU$ PLQRU

6FRUH
PRGHO

0XVLFDOVFRUH
7HPSRUDOGHYLDWLRQ

)
PRGHO
)UHTXHQF\GHYLDWLRQ

9RFDO)

Figure 1: The generative process of a vocal F0 trajectory
based on a hierarchical hidden semi-Markov model involving a score model and an F0 model.
In this study, we tackle musical note estimation for vocal F0 trajectories that tend to have large deviations fromoriginal musical scores. The pitches and onset times ofmusical notes in a musical score can take only discrete values, whereas an F0 trajectory is a continuous signal thatcan dynamically and smoothly vary over time. F0 trajectories are modulated by vibrato and changes smoothly from
one note to another by a portamento. Naive time-frequency
quantization of an F0 trajectory therefore outputs a note sequence that often includes statistically-rare chromatic note
progressions with unlikely rhythms.
To solve this problem, we propose a statistical methodof scale- and rhythm-aware musical note estimation basedon integration of a score model describing the process ofgenerating a note sequence and an F0 model describing theprocess of generating an F0 trajectory from the note sequence (Fig. 1). In the score model, a sequence of musicalscales (local keys) is determined by a Markov process andthe semitone-level pitch of each note is then determined according to both a scale of the note position and the pitch ofa previous note. The onset position of each note on a tatum
grid is determined according to that of a previous note
to make rhythmic structures. In the F0 model, the timefrequency deviations are added to a step-function-shaped
F0 trajectory corresponding to a musical score given by
the score model. The integrated model is thus formulated
as a hierarchical hidden semi-Markov model (HHSMM).
Given a vocal F0 trajectory with a tatum grid, the scales,musical notes, and F0 deviations, which are all latent variables of the proposed model, are jointly estimated by us-ing a Markov chain Monte Carlo algorithm. A key featureof our method is that musical scales and rhythms work as
self-organizing constraints on time-frequency quantizationof vocal F0 trajectories.
2. RELATED WORKIn this section, we introduce related work on the analysis
of singing voices.
2.1 Vocal F0 Estimation for Music Audio Signals
Estimation of vocal F0 trajectories for music audio signals
has actively been studied [1, 3, 7, 9, 14], and the outputsof these methods can be used as inputs of our method.
One of the most basic method is subharmonic summation(SHS) [7] that calculates the sum of the harmonic components of each candidate F0. Ikemiya et al. [9] improved
F0 estimation based on SHS and singing voice separation
based on robust principle component analysis (RPCA) [8]by using the mutual dependency of those two tasks. Salamon et al. [21] estimated contours of the melody F0 candidates by calculating a salience function and then recursively removed contours which do not form a melody line
by using the characteristics of each contour. Durrieu etal. [3] extracted a main melody by representing accompaniments with a model inspired by non-negative matrix factorization (NMF) and leading voices with a source-filter
model. Mauch et al. [14] modified the YIN [1] in a probabilistic way so that the modified system could determine
multiple candidate fundamental frequencies and then select one at each frame by using an HMM.
2.2 Musical Note Estimation for Singing Voices
Estimation of musical notes from sung melody has beena hot research topic [6, 11, 13, 15, 17, 18, 20, 23]. A naive
method is to take the majority of vocal F0s in each intervalof a regular grid [6]. Paiva et al. [17] proposed a cascading method based on multipitch detection, multipitch trajectory construction, segmentation of multipitch trajectory,
elimination of irrelevant notes, and extraction of notes thatform a main melody. Raphael [18] proposed an HMMbased method that estimates pitches, rhythms, and tempos
when the number of notes is given. The rhythm and onset deviation models used in [18] are similar to those usedin our method. Laaksonen et al. [11] divided audio data
into segments corresponding to scales and notes by focusing on the boundaries of chords given as input, and independently estimated the notes based on a score function.
Ryynänen et al. [20] proposed a method based on a hierarchical HMM in order to capture the different kinds of vocal
fluctuations (e.g., vibrato and portamento) within one note.
In this model, the transition between pitches is represented
in the upper-level HMM and the transition between the
vocal fluctuations is represented in the lower-level HMM.
Molina et al. [15] focused on the hysteresis characteristics of vocal F0s. Nishikimi et al. [16] proposed a method
based on an HHM that represents the generative process ofa vocal F0 trajectory considering the time and frequency
deviations. Yang et al. [23] proposed a method based ona hierarchical HMM that represents the generative processof the f0 -∆f0 plane. Mauch et al. [13] developed a software tool called Tony for extracting pitches. In this tool, a
vocal F0 trajectory is estimated by PYIN [14], and musicalnotes are estimated by a modified version of Ryynänen’s
method [20].
3. PROPOSED METHODThis section explains the proposed method of estimating asequence of musical notes from a vocal F0 trajectory. The
method is based on an HHSMM (Fig. 1) that stochasticallygenerates the F0 trajectory with time-frequency deviations
from a sequence of musical notes depending on musicalscales. The upper part of the proposed model is an HMM
that stochastically generates a sequence of musical notes
according to the scales that are assigned to bars. The lower
part is an HSMM that represents the musical notes andtemporal deviations as latent variables and the frequency
deviations as F0 emission probabilities.
3.1 Problem Specification
The problem we tackle is defined as follows:
T

Input: A vocal F0 trajectory X= {xt }t=1 and 16th-noteN
level tatums Y = {(un , vn )}n=0 ,J
Output: A sequence of notes Z= {zj =(pj , lj )}j=0 ,where T is the number of frames in a vocal F0 trajectory, xt
is a log frequency at time t, and N is the number of 16thnote-level tatums. un ∈ {1, . . ., T +1} is the time of tatumn and the beginning and end of music are represented as
u0 = 1 and uN = T +1, respectively. vn ∈ {0, . . ., 15} isthe relative position of tatum n in a bar. J is the number ofmusical notes estimated by proposed methods, and the j-thnote zj is represented as a pair consisting of an pitch pj ∈{1, . . . , K} and a note length lj ∈ {1, . . . , L} in the unit oftatums, where K is the number of kinds of semitone-level
pitches, and pj indicates any one in {µ1 , . . . , µK }, which
is a set of log frequencies corresponding to semitone-level
pitches. For convenience we introduce the initial note z0
that does not appear in the actual score.
3.2 Probabilistic Modeling of Musical ScoresThis section describes the score model constructed with an
HMM that represents rhythms and pitches of musical notes
under musical scales.
3.2.1 Modeling Scale Transitions
M

Scales are represented as S = {sm }m=0 , where M denotes the number of bars in the musical piece and sm denotes the scale at the m-th bar. For convenience, we introduce the initial bar s0 to which the initial note z0 belongs.
Instead of fixing one scale for the whole piece, the scaleis allowed to change at bar lines. Each scale sm takes oneof the 24 values of {C, C#, · · · , B} × {major, minor}. The
latent variables S are described by a Markov chain asp(s0 |π) = πs0 ,(1)p(sm |sm−1 , ξ sm−1 ) = ξsm−1 sm ,(2)24where π∈R≥0 is a set of initial probabilities and ξ s ∈R24≥0
is a set of transition probabilities.

3LWFK>FHQW@& PDMRU݃௝ିଵ‫ݑ‬௡ି௟ೕ

3LWFK

݃௝
݀௝ାଵ

ߤො௧

ߤ௣ೕ

(
'
'
&
&





















     

7DWXP
,QGH[



ߤ௣ೕషభ
7LPH

݈௝

Figure 4: The black bold line represents a sequence of thelocation parameters of the Cauchy distributions.

7UDQVLWLRQEHWZHHQWDWXPVFigure 2: Overview of the score model.
3LWFK

߬௝

‫ݑ‬௡

݀௝

&RQWUROWKHWHQGHQF\RIHDFKSLWFKbVDSSHDUDQFH

7UDQVLWLRQEHWZHHQSLWFKHV߬௝ିଵ

3.3.1 Modeling Temporal Deviations

3LWFK

7DWXPWLPH

We assume that vocal F0 trajectories include the following
two types of temporal deviations (Fig. 3a):

9RFDO)

Onset deviation: the gap between the vocal onset time
and the note onset time.

9RFDO)
2QVHWGHYLDWLRQ)WUDQVLWLRQDOGXUDWLRQ
1RWH

1RWH

)UHTXHQF\GHYLDWLRQ
7LPH

7LPH

(a) Temporal deviations(b) Frequency deviations

Figure 3: Deviations in a vocal F0 trajectory.
3.2.2 Modeling Pitch Transitions
The sequence of pitches P is generated by a Markov chain
depending on scales S as follows (Fig. 2):p(p0 |s0 , ϕs0 ) = ϕs0 p0 ,p(pj |pj−1 , sm , ψ sm pj−1 ) = ψsm pj−1 pj ,(3)ψ̂ŝ deg(pj−1 ;sm )deg(pj ;sm )ψsm pj−1 pj = ∑K m,
p=1 ψ̂ŝm deg(pj−1 ;sm )deg(p;sm )p(gj |ρ) = ρgj ,(8)where ρ ∈ R2G+1
is a set of onset deviation probabilities.
≥0

(5)

(6)

p(dj |η) = ηdj ,D
where η ∈ R≥0 is a set of duration probabilities.

(9)

3.3.2 Modeling Frequency Deviations

where ŝ∈{major,minor} is the mode of scale s and deg(p;s)∈{0, . . . , 11} is the degree of pitch p in scale s (defined as
the relative pitch class of p from the tonic of scale s). ϕ̂∗and ψ̂∗ are the initial and transition probabilities of pitchclasses, given the scales.
3.2.3 Modeling Onset Transitions
Considering the transition between onset positions of adjacent notes, the model makes Z have the plausible rhythm.
N
Let rj−1 ∈ {vn }n=1 be the onset position of the j-th notezj . The transition probability is given byp(rj |rj−1 , ζ rj−1 ) = ζrj−1 rj ,J

The onset deviations G = {gj }j=0 accompanying with
Z are represented as discrete latent variables. Each gj cantake an integer value between −G and G. As with the onset
position model, gj−1 denotes the onset deviation of notezj . We assume that each gj is independently generated by

We assume that there are no deviations for the onset of thefirst note and the offset of the last note, i.e., g0 = gJ = 0.
J
The F0 transitional durations D = {dj }j=1 accompanying with Z are also represented as discrete latent variables. Each dj can take a value from 1 to D. The continuous transition of vocal F0s between notes zj−1 and zj
is represented by a slanted line spanning dj frames. We
assume that each dj is independently generated as follows:(4)

K
where ϕs ∈RK≥0 is a set of initial probabilities, ψ sp ∈R≥0
is a set of transition probabilities, and m is the index ofa bar to which the note zj belongs. Moreover, ϕs0 p0 andψsm pj−1 pj are defined asϕ̂ŝ deg(p0 ;s0 )ϕs0 p0 = ∑K 0,
p=1 ϕ̂ŝ0 deg(p;s0 )F0 transitional duration: the time it takes for singing
voices to finish transitioning from one pitch to the next.

(7)

where the distance between rj−1 and rj indicates the notevalue lj of note zj . We assume that r0 = v0 and rJ = vN .
3.3 Probabilistic Modeling of F0 Trajectories
The section describes the F0 model based on an HSMM
that represents the generative process of a vocal F0 trajectory. In our model, the pitches, onsets, and temporal deviations are represented as latent variables, and the frequency
deviations are represented as emission probabilities.

The vocal F0 trajectory X = {xt }Tt=1 is generated by imparting probabilistic frequency deviations to the sequenceof notes to which probabilistic temporal deviations havealready been imparted (Fig. 3b). Assuming that xt is independently generated at each frame, the emission probability of the j-th note zj is given byp(xτj−1 :τj −1 |pj−1 , pj , lj , gj−1 , gj , dj , µ̂t , λ)
τj −1

=

∏

{δxt ,voiced Cauchy(xt |µ̂t , λ) + δxt ,unvoiced }t=τj−1= epj−1 pj lj gj−1 gj dj ,(10)where xτ ′ :τ −1 indicates xτ ′ , . . . , xτ −1 , λ is a scale parameter that represents the scale of the frequency deviations, δis Kronecker’s delta, and µ̂t (Fig. 4) is a location parameter
given by{µp −µpj
j−1(t−τj−1 )+µpj−1 (τj−1 ≤t<τj +dj )dj. (11)µ̂t =µ pj(τj−1 +dj ≤t<τj )When the onset of note zj+1 is located at the n-th tatum,τj = un + gj and τj−1 = un−lj + gj−1 .

థ෡థ෡ࢻ୫୧୬୭୰ ൌ

αsSm = p(p0:jm+1 −1 , sm )ࢻ୫ୟ୨୭୰ ൌ

jm+1 −1            'HJUHH

=

∏

j=jm

Figure 5: The configuration of the hyperemarameter aϕ̂ŝ .
3.4 Prior Distributions
We put conjugate Dirichlet priors on categorical model parameters π, ξ, ϕ̂, ψ̂, ζ, ρ, and η as follows:( )( )π ∼ Dirichlet aπ , ξ s ∼ Dirichlet aξs ,( )()ϕ̂ŝ ∼ Dirichlet aϕ̂ŝ , ψ̂ ŝdeg(p;s) ∼Dirichlet aψ̂ŝdeg(p;s) ,( )ζ r ∼ Dirichlet aζr ,( )( )ρ ∼ Dirichlet aρ , η ∼ Dirichlet aη ,
(12)
ϕ̂
ψ̂
16
ξ
26
12
12
ζ
where a ∈R26
+ , as ∈R+ , aŝ ∈R+ , aŝdeg(p;s) ∈R+ , ar ∈R+ ,aρ ∈R2G+1, and aη ∈RD+ are hyperparameters. The prob+πability distribution over the 12 pitch classes under a scaleis estimated using the priors on the initial and transitional
probabilities of those classes. As illustrated in Fig. 5, we
set the hyperparameters aϕ̂ŝ and aψ̂ŝdeg(p;s) so that the probability distributions represent the diatonic scales, respectively. Since the Cauchy distribution does not have a conjugate prior, we put a Gamma prior on λ as()
λ ∼ Gamma aλ0 , aλ1 ,(13)
where aλ0 and aλ1 are shape and rate hyperparameters.
3.5 Bayesian Inferenceψsm pj−1 pj∑

ξsm−1 sm αsSm−1 ,

(16)

sm−1

where jm is the index of the first note whose onset belongs
to the m-th bar. jm can be calculated from given note values L.
In backward sampling, Eq. (14) is calculated by using
the values calculated in forward filtering, and scales are
sampled recursively as follows:
βsSM = p(sM |Z) ∝ αsSM ,(17)

βsSm = p(sm |sm+1:M , Z) ∝ αsSm ξsm sm+1 .

(18)

3.5.2 Inferring Latent Variables Q
The latent variables Q can be estimated in a way similar
to that in which the latent variables S are inferred. In
forward filtering, we recursively calculate the probability
αpQn ln ,gn dn as follows:
αpQ0 l0 g0 d0 = p(p0 |S) = ϕy0 p0 ,(19)

αpQn ln gn dn= p(x1:τn −1 , pn , ln , gn , dn |S)0(ln >n)ρgn ηdn ζr0 rnQ ·∑ ψ
p0 s1 p0 pn ep0 pn ln 0gn dn αp0 l0 g0 d0 (ln =n), (20)=′
 ∑ min(n∑ ,L) ∑ρgn ηdn ζrn′ rn ψsm(n′ ) pn′ pnpn′ ,gn′
ln′
dn′



· epn′ pn ln gn′ gn dn αpQn′ ln′ gn′ dn′(ln <n)Given an F0 trajectory X, we aim to calculate the posterior
distribution p(Q, S, Θ|X), where Q = {P , L, G, D}(latent variables) and Θ = {π, ξ, ϕ̂, ψ̂, ζ, ρ, η} (model
parameters). Since this calculation is analytically intractable, we use Markov chain Monte Carlo (MCMC) methods.
To get samples of the latent variables S and Q, forward
filtering-backward sampling algorithms are used. To getsamples of Θ except for λ, a set of parameters with conjugate priors, a Gibbs sampling algorithm is used. Since
there is no conjugate prior for the parameter λ, we use
the Metropolis-Hastings (MH) algorithm. Since S and Q
share the sequence of notes Z and are mutually dependent,
each variable is updated as follows:
1. Initialize notes Z with a majority-vote method.
2. Update the sequence of scales S based on given Z.
3. Update Q based on given S.
4. Update the model parameters Θ.
5. Return to 2.
3.5.1 Inferring Latent Variables S

where τn = un + gn , n′ = n − ln , and m(n′ ) is the index
of the bar that the n′ -th tatum belongs to. pn , ln , gn , anddn are the variables of forward messages that correspondto the note whose offset position is at the n-th tatum un .
Note that these variables are different from j-indexed variables pj , lj , gj , and dj . Since the onset and offset timesof the note zn = (pn , ln ) are respectively the (n−ln )-thtatum and the n-th tatum, the probability p(ln ) which appears in the recursive calculation of Eq. (20) is replaced byp(rn |rn−ln ).
In backward sampling, the posterior distribution of thelatent variables is calculated by using the values calculated
in forward filtering, and notes and temporal deviations are
sampled recursively as follows:

Given the sequence of notes Z, each sm is sampled in accordance with the probability given by(21)

βsSm = p(sm |sm+1:M , Z),(14)

where sm+1:M represents sm+1 , . . . , sM . The calculationof Eq. (14) and sampling of scales S are performed by theforward filtering-backward sampling method.
In forward filtering, we recursively calculate the probability αsSm as follows:
αsS0 = p(p0 , s0 ) = p(p0 |s0 )p(s0 ) = ϕs0 p0 πs0 ,(15)βpN lN gN dN = p(pN , lN , gN , dN |X, S) ∝ αpQN lN gN dN ,βpn′ ln′ gn′ dn′= p(pn′ , ln′ , gn′ , dn′ |pn:N , ln:N , gn:N , dn:N , X)
(ln >n)
0∝ epn′ pn ln gn′ gn dn ψsm(n′ ) pn′ pn.
Q· ζrn′ rn ρgn ηdn αpn′ ln′ gn′ dn′ (ln ≤ n)
3.5.3 Learning Model Parameters ΘThe posterior distributions of the model parameters withthe prior distributions are calculated using S and Q obtained in the backward sampling steps, and these parameters are sampled according to the posterior distributions asfollows:()()
π∼Dirichlet aπ +bπ , ξ s ∼Dirichlet aξs +bξs , (22)()ϕ̂ŝ ∼Dirichlet aϕ̂ŝ +bϕ̂ŝ ,()ψ̂ψ̂ ŝdeg(p;s) ∼Dirichlet aψ̂ŝdeg(p;s) +bŝdeg(p;s) ,()ζ r ∼Dirichlet aζr +bζr ,()()ρ ∼Dirichlet aρ +bρ , η∼Dirichlet aη +bη ,

(23)
(24)
(26)

(27)

∗

where λ is a proposal, λ is the current sample, and γ isa hyperparameter. The proposal λ∗ is accepted as the next
sample according to the probability given by{}L(λ∗ )q(λ|λ∗ )∗A(λ , λ) = min,(28)
L(λ)q(λ∗ |λ)where L(λ) is the complete joint likelihood of λ given byJ()∏
L(λ) = Gamma λ|aλ0 , aλ1
epj−1 pj lj gj−1 gj dj , (29)j=1
J

{pj , lj , gj , dj }j=0 are the values sampled in the backward
sampling. The value of λ is updated by λ∗ only when thevalue of A(λ∗ , λ) is larger than a random number sampledfrom the uniform distribution U(0, 1).
3.6 Viterbi Decoding
The sequence of latent variables S and Q are estimatedwith the Viterbi algorithm with the model parameters thatmaximize the joint distribution p(X, Q, S, Θ|Φ) in thelearning process. As in the inference of latent variables, we
initialize Z by the majority-vote method, S is estimatedbased on Z, and then Q is estimated depending on the S
estimated in the previous step.
In the Viterbi decoding on scales S, the value ωsS isrecursively calculated as follows:ωsS0 = ln ϕs0 k0 + ln πs0 ,

(30)

}
{
ln ψsm pj−1 pj +max ln ξsm−1 sm+ωsSm−1 . (31)jm+1 −1

ωsSm=

∑

j=jm

sM = arg max αsSM , sm−1 = cSsm .

sm−1

In the recursive calculation of ωsS , the previous state sm−1

(32)

sM

(25)

where bπ ∈R26
≥0 is a unit vector whose s0 -th element is 1.
ξ26
bs ∈R≥0 is a vector whose s′ -th element indicates the number of transitions between adjacent scales s and s′ in the sequence of latent variables Y . bρ ∈R2G+1
is a vector whose≥0g-th element indicates the number of vocal onset deviations of g in sampled Q, and bη ∈RD≥0 is a vector whosed-th element represents the number of F0 transitional du′rations of d in sampled Q. bζr ∈R16
≥0 is a vector whose r -thelement represents the number of transitions between adJ
jacent note onset positions r and r′ in R= {rj }j=0 thatcan be calculated from the note values L sampled in backward sampling. Regarding the vector bϕ̂ŝ ∈R12
>0 , when thescale of the initial bar and the pitch of the initial note ares0 = s and p0 = p, the value of the element bϕ̂ŝdeg(p;s)
is 1, and the other elements are 0. Regarding the vector
ψ̂
12
bψ̂
ŝdeg(p;s) ∈R≥0 , the value of bŝdeg(p;s)deg(p′ ;s) is increased
by one when there is a transition from a pitch p to a pitchp′ under a scale s in the sampled latent variables.
To apply the MH sampling to the parameter λ, we definea random-walk proposal distribution as follows:q(λ∗ |λ) = Gamma(γλ, γ),that maximizes the value of ωsSm is memorized as cSsm , andthe scales S are recursively estimated as follows:

Q
In the Viterbi decoding on variables Q, the value ωplgd
is recursively calculated as follows:ωpQ0 l0 g0 d0 = wϕ ln ϕs0 p0 ,(33)

ωpQn ln gn dn(ln >n)− infρηζw ln ρgn +w
{ ln ηdn +w ln ζrn r0
 + maxψ
p0 w ln ψs1 p0 pn}+ we ln ep0 pn ln 0gn dn + ωpQ0 l0 g0 d0(ln =n) ,
=


ρ
η
ζ

w ln ρgn +w ln ηdn +w{ ln ζrn rn′






+ max(pn′ ,ln′ ,gn′ ,dn′ ) wψ ln ψsm(n′ ) pn′ pn


}


 + we ln ep p l g g d + ω Q(ln <n)pn′ ln′ gn′ dn′n′ n n n′ n n
(34)
ϕ

ψ

ρ

η

ζ

e

where w , w , w , w , w , and w are the weight parameters that control the balance between probabilities. In
Q
the recursive calculation of ωplgd, the previous states pn′ ,ln′ , gn′ , and dn′ which maximize the value of ωpQn ln gn dn
are memorized as cQ
pn ln gn dn , and the variables Q are recursively estimated as follows:(pN , lN , gN , dN ) = arg max αpQN lN gN dN ,(35)(pn′ , ln′ , gn′ , dn′ ) = cQ
pn ln gn dn .

(36)

pN ,lN ,gN ,dN

4. EVALUATION
We report comparative experiments conducted to evaluate
the performance of the proposed method in musical note
estimation from vocal F0 trajectories.
4.1 Experimental Conditions
Among the 100 pieces of popular music in the RWC music database [5], we used 63 pieces that do not include
32nd notes, triplets, harmonizing parts, and overlaps of adjacent notes, which the proposed method cannot deal with.
The input F0 trajectories were obtained from the annotation data [4] or automatically estimated by using the stateof-the-art melody extraction method proposed in [9]. The
annotation data contain unvoiced regions and the estimation data do not. The tatum times and onset positions wereobtained from the annotation data.
The Bayesian inference and Viterbi decoding were independently conducted for each song. The onset transition probabilities were learned in advance from a corpus
of rock music [2] without Bayesian learning. The hyperparameters were aπ =1, aξs =1, aζr =1, aρ = aη = aλ0 =aλ1 = γ =1, where 1 and 1 respectively represent the matrix and vector whose elements are all ones. The elementsof aϕ̂ŝ and aψ̂ŝdeg(p;s) corresponding to musical notes on thescale of ŝ were 10 and the others were 1. The weight parameters of the Viterbi algorithm were empirically set as
wϕ = wψ = 29.4, wρ = 2.4, wη = 2.9, wζ = 48.5, and
we = 3.8. To obtain musically-consistent sequences ofModelInput F0s

Tatum level Note level

Proposed
Ground-truth 72.4 ± 1.7
methodEstimated
68.7 ± 1.3
WithGround-truth 71.5 ± 1.6
only rhythms Estimated
67.7 ± 1.3
WithGround-truth 67.8 ± 1.6
only scales
Estimated
65.6 ± 1.2
Without scales Ground-truth 67.2 ± 1.5& rhythms
Estimated
64.6 ± 1.2
Majority vote Ground-truth 54.1 ± 1.5
Estimated
61.0 ± 1.4
HMM [16]
Estimated
68.0 ± 1.2

28.1 ± 2.1
30.7 ± 1.8
26.3 ± 2.1
29.1 ± 1.8
10.6 ± 1.2
13.8 ± 1.1
9.8 ± 1.2
12.9 ± 1.1
20.1 ± 1.4
22.0 ± 1.5
14.8 ± 1.3

Table 1: Average matching rates [%] and their standard
errors in tatum and note levels.
musical notes, we put more emphasis on the score model
than the F0 model.
For comparison, we tested the majority-vote method asa baseline and the latest conventional method based ona semi-beat-synchronous HMM [16]. Since the conventional method cannot deal with unvoiced regions in a vocal
F0 trajectory given as input, we only tested the method
for the estimation data. To evaluate the effectiveness ofthe score model, we tested four versions of the proposed
method; a method that does not consider scales (scale transition probabilities) and rhythms (onset transition probabilities), a method considering only scales, a method considering only rhythms, the full method considering bothscales and rhythms. To accelerate the inference, the searchrange of pitches was limited around the pitches estimatedby the majority-vote method.
To evaluate the performance of each method, we calculated tatum-level and note-level matching rates by comparing the estimated sequences of musical notes with theground-truth data. The tatum-level matching rate is the rateof the number of tatum units whose pitches were estimatedcorrectly to the total number of tatum units whose pitches
exist in the ground-truth scores. The note-level matching rate is the rate of the number of musical notes whose
pitches, onsets, and offsets were estimated correctly to thetotal number of musical notes in the ground-truth scores.
If adjacent notes in the ground-truth scores have the same
pitch or are connected by a tie, those notes were regarded
as a single note. Since the compared method [16] outputs
a pitch in a 16th-note-wise manner, a sequence of the same
pitches was regarded as a single note.
4.2 Experimental Results
The experimental results are shown in Table 11 . The proposed method outperformed the majority-vote method andthe conventional method in terms of both measures. Comparing the tatum-level matching rates obtained by the four
versions of the proposed method, we confirmed that thescore model improved the performance of musical note
estimation. The use of the onset transition probabilities
1 The results of music note estimation by the proposed method areavailable online: http://sap.ist.i.kyoto-u.ac.jp/members/nishikimi/demo/
ismir2017/&RUUHFWVFRUH(VWLPDWHGVFRUH(VWLPDWHGVFRUH ZLWKRXWWKHVFDOH UK\WKPPRGHOVFigure 6: Musical scores estimated from a ground-truth F0
trajectory by the proposed method and its variant withoutscale and rhythm constraints.
(rhythm constraints) was found to be more effective thanthat of the scale transition probabilities (scale constraints).
Although the tatum-level matching rate obtained the proposed method (68.7%) was close to that obtained by theconventional method (68.0%), the note-level matching rate
obtained the proposed method (30.7%) was better than thatobtained by the conventional method (14.8%), This is a remarkable advantage of the proposed HHSMM that can directly represent both the pitches and durations (onsets andoffsets) of musical notes on symbolic musical scores, noton continuous-time piano rolls.
Examples of estimated musical scores are illustrated inFig. 6. The proposed method yielded the almost accuratemusical score except that some notes were merged. To correctly recognize two adjacent notes with the same pitch, it
is necessary to refer to original singing voices or music audio signals. The score estimated without considering thescore model, on the other hand, included a lot of wrong
notes that were inconsistent with music theory. This result
also shows the effectiveness of using the score model asmusical constraints on musical note estimation.
5. CONCLUSION
This paper presented a statistical method for musical note
estimation from a vocal F0 trajectory. Our method is basedon an HHSMM that combines a score model (HMM) representing the generative process of a musical score frommusical scales with an F0 model (HSMM) representing
the generative process of a vocal F0 trajectory with timefrequency deviation from the musical score. We confirmedthat the proposed method can yield more musically-consistent
sequences of musical notes.
One of the most interesting directions of this research is
to use the proposed model as a musically-meaningful priordistribution on a vocal F0 trajectory in vocal F0 estimation
for music audio signals. We plan to integrate the proposed“language” model that generates an F0 trajectory from amusical score with an acoustic model that generates a spectrogram from the F0 trajectory in a hierarchical Bayesianmanner. This enables us to jointly learn the vocal F0 trajectory and musical score from music audio signals. Joint
estimation of beat times and F0s is worth investigating toovercome the problem of estimation-error accumulation in
the cascaded estimation approach.
Acknowledgement: This study was partially supported by JSPS
KAKENHI Grant Numbers 26700020, 16H01744, and 16J05486
and JST ACCEL No. JPMJAC1602.

6. REFERENCES[1] A. de Cheveigné and H. Kawahara. YIN, a fundamental frequency estimator for speech and music. The Journal of the Acoustical Society of America, 111(4):1917–
1930, 2002.
[2] T. De Clercq and D. Temperley. A corpus analysis ofrock harmony. Popular Music, 30(01):47–70, 2011.
[3] J.-L. Durrieu, G. Richard, B. David, and C. Févotte.
Source/filter model for unsupervised main melody extraction from polyphonic audio signals. IEEE Transactions on Audio, Speech, and Language Processing,18(3):564–575, 2010.
[4] M. Goto. Aist annotation for the RWC music database.
In The 7th International Conference on Music Information Retrieval (ISMIR 2006), pages 359–360, 2006.

[13] M. Mauch, C. Cannam, R. Bittner, G. Fazekas, J Salamon, J. Dai, J. Bello, and S Dixon. Computer-aided
melody note transcription using the Tony software: Accuracy and efficiency. In Proc. of the 1st International
Conference on Technologies for Music Notation and
Representation (TENOR 2015), pages 23–30, 2015.
[14] M. Mauch and S. Dixon. pYIN: A fundamental frequency estimator using probabilistic threshold distributions. In 2014 IEEE International Conference on
Acoustics, Speech and Signal Processing (ICASSP
2014), pages 659–663, 2014.
[15] E. Molina, L. J. Tardón, A. M. Barbancho, and I. Barbancho. Sipth: Singing transcription based on hysteresis defined on the pitch-time curve. IEEE/ACM Transactions on Audio, Speech and Language Processing(TASLP), 23(2):252–263, 2015.

[5] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka.
RWC music database: Popular, classical and jazz music databases. In The 3rd International Conference onMusic Information Retrieval (ISMIR 2002), pages 287–
288, 2002.

[16] R. Nishikimi, E. Nakamura, K. Itoyama, and K Yoshii.
Musical note estimation for f0 trajectories of singing
voices based on a bayesian semi-beat-synchronoushmm. In Proc. of the 17th International Society for Music Information Retrieval Conference (ISMIR 2026),
pages 461–467, 2016.

[6] M. Goto, K. Yoshii, H. Fujihara, M. Mauch, andT Nakano. Songle: A web service for active music
listening improved by user contributions. In Proc. of
the 12th International Society for Music Information
Retrieval Conference (ISMIR 2011), pages 311–316,
2011.

[17] R. P. Paiva, T. Mendes, and A. Cardoso. On the detection of melody notes in polyphonic audio. In 6th International Conference on Music Information Retrieval(ISMIR 2005), pages 175–182, 2005.

[7] Dik J. Hermes. Measurement of pitch by subharmonicsummation. The journal of the acoustical society ofAmerica, 83(1):257–264, 1988.
[8] P.-S. Huang, S. D. Chen, P. Smaragdis, and
M. Hasegawa-Johnson. Singing-voice separation frommonaural recordings using robust principal component analysis. In 2012 IEEE International Conferenceon Acoustics, Speech and Signal Processing (ICASSP
2012), pages 57–60, 2012.
[9] Y. Ikemiya, K. Yoshii, and K. Itoyama. Singing voice
analysis and editing based on mutually dependent F0
estimation and source separation. In 2015 IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP 2015), pages 574–578, 2015.
[10] Y. E. Kim and B. Whitman. Singer identification in
popular music recordings using voice coding features.
In 3rd International Conference on Music Information
Retrieval (ISMIR 2002), volume 13, page 17, 2002.
[11] A. Laaksonen. Automatic melody transcription basedon chord transcription. In Proc. of the 15th International Society for Music Information Retrieval (ISMIR
2014), pages 119–124, 2014.
[12] Y. Li and D. Wang. Separation of singing voice frommusic accompaniment for monaural recordings. IEEE
Transactions on Audio, Speech, and Language Processing, 15(4):1475–1487, 2007.

[18] C. Raphael. A graphical model for recognizing sung
melodies. In 6th International Conference on Music
Information Retrieval (ISMIR 2005), pages 658–663,
2005.
[19] M. Ryynänen, T. Virtanen, J. Paulus, and A. Klapuri. Accompaniment separation and karaoke application based on automatic melody transcription. In
2008 IEEE International Conference on Multimedia
and Expo, pages 1417–1420, 2008.
[20] M. P. Ryynänen and A. P. Klapuri. Automatic transcription of melody, bass line, and chords in polyphonic music. Computer Music Journal, 32(3):72–86,
2008.
[21] J. Salamon and E. Gómez. Melody extraction frompolyphonic music signals using pitch contour characteristics. IEEE Transactions on Audio, Speech, andLanguage Processing, 20(6):1759–1770, 2012.
[22] W.-H. Tsai and H.-M. Wang. Automatic singer recognition of popular music recordings via estimation andmodeling of solo vocal signals. IEEE Transactions on
Audio, Speech, and Language Processing, 14(1):330–
341, 2006.
[23] L. Yang, A. Maezawa, J. B. L. Smith, and E. Chew.
Probabilistic transcription of sung melody using a pitch
dynamic model. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP 2017), pages 301–305, 2017.