CONTEXT-FREE 2D TREE STRUCTURE MODEL OF MUSICAL NOTESFOR BAYESIAN MODELING OF POLYPHONIC SPECTROGRAMS
Hirokazu Kameoka1,2 , Kazuki Ochiai1 , Masahiro Nakano2 , Masato Tsuchiya1 , Shigeki Sagayama1
1
Graduate School of Information Science and Technology, The University of Tokyo
Hongo 7-3-1, Bunkyo, Tokyo, 113-8656, Japan
2
NTT Communication Science Laboratories, NTT Corporation
Morinosato Wakamiya 3-1, Atsugi, Kanagawa, 243-0198, Japan

ABSTRACTThis paper proposes a Bayesian model for automatic music transcription. Automatic music transcription involves
several subproblems that are interdependent of each other:
multiple fundamental frequency estimation, onset detection, and rhythm/tempo recognition. In general, simultaneous estimation is preferable when several estimation problems have chicken-and-egg relationships. This paper proposes modeling the generative process of an entire music
spectrogram by combining the sub-process by which a musically natural tempo curve is generated, the sub-process
by which a set of note onset positions is generated basedon a 2-dimensional tree structure representation of music,
and the sub-process by which a music spectrogram is generated according to the tempo curve and the note onset positions. Most conventional approaches to music transcription perform note extraction prior to structure analysis, but
accurate note extraction has been a difﬁcult task. By contrast, thanks to the combined generative model, the present
method performs note extraction and structure estimation
simultaneously and thus the optimal solution is obtained
within a uniﬁed framework. We show some of the transcription results obtained with the present method.
1. INTRODUCTION
Music transcription is the process of automatically converting a given audio signal into a musical score. Although
there are a number of viable ways of transcribing monophonic music, polyphonic music still poses a formidable
challenge.
Several subproblems must be solved if we are to transcribe polyphonic music automatically, namely source separation, multiple fundamental frequency estimation (the estimation of the fundamental frequencies of concurrent musical sounds), onset detection (the detection of the positionin the signal where each note begins), and rhythm recognition (the estimation of the tempo, beat locations, and thenote value of each note). The difﬁculty is that these subproblems involve many ambiguities.
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for proﬁt or commercial advantage and that copies
bear this notice and the full citation on the ﬁrst page.
c 2012 International Society for Music Information Retrieval.
⃝

An audio signal of a musical note typically consists ofmany overtones, some of which usually overlap when multiple notes are played simultaneously. To detect whichnotes are present at a certain time instant, we need to know
which musical note each frequency component belongs to.
Since this information is missing for the spectrum of a mixture signal, there can be multiple interpretations of how the
spectrum of each sound should appear as well as which
pitches are present in the mixture. On the other hand, amusic performance often involves temporal ﬂuctuation interms of both rhythm and tempo, which means performers
do not always play notes with a perfectly timed rhythm andconstant tempo. Since we cannot deﬁne a note value without having a notion for tempo and vice versa, there can beinﬁnite interpretations regarding what the intended rhythm
was and how the tempo varied if both types of information
are missing.
Many methods have already been developed for polyphonic music transcription, most of which try to tackle theproblem by dealing with the abovementioned subproblems
separately [1]. However, the inherent difﬁculty of the music transcription problem lies in the chicken-and-egg interdependency between these subproblems [2]. Firstly, if thegiven signal is already decomposed into individual notes, it
is a simple matter to detect their fundamental frequencies.
On the other hand, the decomposition of a given spectrogram into individual notes can be accomplished more accurately when the fundamental frequencies of the concurrent
sounds are given. Also, if we know the fundamental frequencies of all the underlying notes in the signal, they canconstitute very useful information for accurately estimating their onset times and vice versa. Furthermore, as theonset times of notes are usually governed by the rhythmic
structure of a piece of music, the “chicken and egg” situation also applies to the detection of note onsets and thedetermination of beat locations and tempo. If we know thebeat locations of a piece of music, then it is much easier
to detect the onset times of notes and vice versa, since theinter-onset times are likely to be multiples or fractions of
the beat period.
Simultaneous estimation is generally preferable whenseveral estimation problems are interdependent. Thus, weconsider it necessary to introduce a uniﬁed model, which
could be used to jointly solve the problems of determining
the pitch and onset time of each musical note, the rhythm
and the overall tempo variation of a piece of music. In thispaper, we take a Bayesian approach (a generative modelapproach) as in [3–6] to formulate and solve this simultaneous estimation problem.
2. GENERATIVE MODEL OF SPECTROGRAM
2.1 Overview
Motivated by the above, this paper proposes modeling thegenerative process of an entire spectrogram of a piece ofmusic by formulating the following three sub-processes
and combining them into one process: (1) the sub-process
by which the tempo curve of a piece of music is generated, (2) the sub-process by which a set of note onset positions (in terms of the relative time) is generated basedon a 2-dimensional tree structure representation of music,
and (3) the sub-process by which a music spectrogram isgenerated according to the tempo curve generated by subprocess 1 and the set of note onset positions generated bysub-process 2. In the following, we model sub-process 1 in
2.2, sub-process 2 in 2.3 and sub-process 3 in 2.4, respectively. Our aim is to use this complete generative model toexplain how a given spectrogram is generated. The most
likely model parameters given the observation would thengive a musically likely interpretation of what is actually
happening in the spectrogram (i.e., a musical score). To
this end, we employ a Bayesian approach to infer the posterior distributions of all the latent parameters. An approximate posterior inference algorithm is derived, which is described in Section 3.
2.2 Sub-process for generating tempo curve
The tempo of a piece of music is not always constant and in
most cases it varies gradually over time. If we use a 1 “tick”
as a relative time notion, an instantaneous (or local) tempo
may be deﬁned as the length of 1 tick in seconds. Now
let us use µd to denote the real duration (in units of seconds) corresponding to the interval between d and d + 1
ticks. Thus, µd corresponds to the local tempo and so thesequence µ1 , . . . , µD can be regarded as the overall tempo
curve of a piece of music. One reasonable way to ensure
a smooth overall change in tempo is to place a Markovchain prior distribution over the sequence µ1 , . . . , µD thatis likely to generate a sequence µ1 , . . . , µD such that µ1 ≃µ2 , µ2 ≃ µ3 , . . . , µD−1 ≃ µD . Here, we assume a Gaussianchain prior for convenience:µd |µd−1 ∼ N (µd ; µd−1 , (σ µ )2 ) (d = 2, . . . , D),(1)(x−µ)2where N (x; µ, σ) ∝ e− 2σ2 . If we use ψd to denotethe absolute time (in units of seconds) corresponding tod ticks, ψd can thus be written as ψd = ψd−1 + µd−1 ,which plays the role of mapping a relative time in unitsof ticks (integer) to an absolute time in units of seconds(continuous value).
2.3 Sub-process for generating note onset positions
Here we describe the generative model of the set of somenumber R of note onset positions S1 , . . . , SR (in units ofticks). Most people would probably agree that music has
1 Tick is a relative measure of time represented by the number of discrete divisions a quarter note has been split into. So, if we consider 16
divisions per quarter note, for instance, the duration of 40 ticks corresponds to two-and-a-half beats.

Figure 1. Generative model of a 2-dimensional tree structure representation of musical notes.
a 2-dimensional hierarchical structure. Frequent motifs,
phrases or melodic themes consist of a hierarchy that canbe described as time-span trees. In addition, polyphony often consists of multiple independent voices. That is, we
can assume that music consists of a time-spanning tree
structure and a synchronizing structure of multiple eventsat several levels of a hierarchy. We would like to describe
this 2-dimensional tree structure representation of music in
the form of a generative model. This can be accomplishedby introducing a generative model that is conceptually similar to the one proposed in [6].
Fig. 1 shows an example of the generative process offour musical notes in one bar of 4/4. In this example, awhole note is ﬁrst split into two consecutive half notes.
We call this process “time-spanning.” Next, the former
half note is copied in the same location, thus resulting ina chord of two half notes. We call this process “synchronization.” A chord with an arbitrary number of notes canthus be generated by successively employing this type ofbinary production. Finally, the latter half note is split into aquaver and a dotted quarter note via the time-spanning process. This kind of generative process can be modeled byextending the idea of the probabilistic context-free grammar (PCFG) [7]. For simplicity, this paper focuses only on
Chomsky normal form grammars, which consist of only
two types of rules: emissions and binary productions. A
PCFG is a pair consisting of a context-free grammar (aset of symbols and productions of the form A → BC or
A → w, where A, B, and C are called “nonterminal symbols” and w is called a “terminal symbol”) and production
probabilities, and deﬁnes a probability distribution over thetrees of symbols. The parameters of each symbol consistof (1) a distribution over rule types, (2) an emission distribution over terminal symbols, and (3) a binary production
over pairs of symbols.
To describe the generative process shown in Fig. 1,
we must introduce an extension of PCFG. As we explain
later, we explicitly incorporate a process of stochasticallychoosing either “time-spanning” or “synchronization” inthe binary production process. Fig. 2 deﬁnes the proposed generative process of the set of the onset positionsof some number R of musical notes. In our model, eachnode n of the parse tree corresponds to one musical note(with no pitch information) and a pair consisting of the onset position Sn and duration Ln of that note is considered
to be a nonterminal symbol. We ﬁrst draw a “switching”
distribution (namely, a Bernoulli distribution) ϕT over thetwo rule types {EMISSION, BINARY-PRODUCTION} from a
Beta distribution. Next, we draw another “switching” dis-Draw rule probabilities:
ϕT ∼ Beta(ϕT ; 1, β T )[Probability of choosing either of two rule types]

ϕN ∼ Beta(ϕN ; 1, β N )[Probability of choosing either of two binary-production types]

For each duration l:BB
ϕBl ∼ Dirichlet(ϕl ; β l )[Probability of position at which segment of length l is split]

For each node n in the parse tree:bn ∼ Bernoulli(bn ; ϕT )[Choose either EMISSION or BINARY-PRODUCTION]

If bn = EMISSION
Sr ∼ δSr ,Sn , Lr ∼ δLr ,Ln[Emit terminal symbol]

If bn = BINARY-PRODUCTIONρn ∼ Bernoulli(ρn ; ϕN )[Choose either SYNCHRONIZATION or TIME-SPANNING]

If ρn = SYNCHRONIZATION
Sn1 ∼ δSn1 ,Sn , Sn2 ∼ δSn2 ,Sn
Ln1 ∼ δLn1 ,Ln , Ln2 ∼ δLn2 ,Ln[Produce two copies of note n]

If ρn = TIME-SPANNING
Sn1 ∼ δSn1 ,Sn , Sn2 ∼ δSn2 ,Sn +Ln1
Ln1 ∼ δLn1 ,Ln −Ln2
Ln2 ∼ Discrete(Ln2 ; ϕB
Ln )[Split note n into two consecutive notes n1 and n2 ]Figure 2. The probabilistic speciﬁcation of the present
generative model of a 2-dimensional tree structure representation of musical notes. δ denotes Kronecker’s
delta. Thus, x ∼ δx,y means x = y (with probability 1). Bernoulli(x; y) and Beta(y; z) are deﬁned
as Bernoulli(x; y) = y x (1 − y)1−x and Beta(y; z) ∝y z1 −1 (1 − y)z2 −1 , where x ∈ {0, 1}, 0 ≤ y ≤ 1 and z =(z1 , z2 ), respectively. Discrete(x; y) and Dirichlet(y; z)
areas Discrete(x; y) = yx and Dirichlet(y; z) ∝∏ deﬁned
zi −1
y
where
y = (y1 , . . . , yI ) with y1 + · · · + yI = 1
i i
and z = (z1 , . . . , zI ), respectively.
tribution ϕN over the two binary-production types {TIMESPANNING, SYNCHRONIZATION} similarly from a Beta distribution. Finally, we generate a discrete distribution ϕB
l =
B
′
(ϕB
,
.
.
.
,
ϕ
)
over
the
position
l
at
which
the
segment
l,1l,l
of duration l is split when BINARY-PRODUCTION is chosen.
The shapes of all the Beta distributions and the Dirichlet
distribution in our model are governed by concentrationB
hyperparameters: β T , β N and β B
1 , . . . , βD .
Given a grammar, we generate a parse tree in the following manner: start with a root node that has the designated root symbol, SRoot = 0 and LRoot = D where D
denotes the overall length of a piece of music in ticks. For
each nonterminal node n, we ﬁrst choose a rule type bn using ϕT . If bn = EMISSION, we produce a terminal symbol
Sr with the value of Sn , namely the onset position of noter. If bn = BINARY-PRODUCTION, we then choose a binaryproduction type ρn using ϕN . If ρn = SYNCHRONIZATION,

we produce two nonterminal children n1 and n2 such that
Sn1 = Sn2 = Sn , Ln1 = Ln2 = Ln . This means that thenotes of the child nodes have exactly the same onset andduration. If ρn = TIME-SPANNING, we produce two nonterminal children n1 and n2 with Sn1 = Sn , Ln1 = Ln −Ln2 ,Sn2 = Sn + Ln1 where Ln2 is drawn from a discrete distribution ϕB
Ln . Ln2 corresponds to the position at which
the segment of duration Ln is divided. We apply the procedure recursively to any nonterminal children and ﬁnallyobtain a sequence S1 , . . . , SR corresponding to the onset
positions of R musical notes.
None of the notes r yet contains pitch information. We
assign a pitch index κr to each note r in the same way as
an ordinary cluster assignment process:KK
ϕKr ∼ Dirichlet(ϕr ; α ),(2)κr ∼ Discrete(κr ; ϕK
r ),

(3)

where Discrete(x; y) = yx (where y = (y∏1 , . . . , yI ) withy1 +· · ·+yI = 1) and Dirichlet(y; z) ∝ i yizi −1 (wherez = (z1 , . . . , zI )). The k-th element of ϕKr deﬁnes how
likely each pitch index is to be chosen. It should be noted
here that the generative processes of Sr and κr should not
be considered independently, since harmony and rhythm
are in general interdependent of each other. An interesting
direction for future work is the joint modeling of these twogenerative processes.
2.4 Sub-process for generating spectrogram
We now turn to describing the sub-process by which a music spectrogram is generated. Here, we consider that a music spectrogram is generated according to the tempo curve
and the set of note onset positions, that have been generated by the sub-processes described in 2.2 and 2.3. To
model a spectrogram of a musical audio signal, we makethe following assumptions about musical notes:(A1) Each musical note has a static spectral proﬁle characterized by a particular pitch.
(A2) The magnitude spectrum of music at a certain time
instant is represented by a superposition of the spectra of multiple musical notes.
(A3) The power of each musical note varies smoothly intime in the interval between the onset and offset.
From assumption (A1), a magnitude spectrogram of a musical note r can be described as
Xω,t =

R∑

Hω,κr Wr,t ,

(4)

r=1

where ω and t are frequency and time indices, respectively.
A set consisting of H1,k , . . . , HΩ,k ≥ 0 represents thestatic spectrum of the k-th pitch and so a set consisting of
H1,κr , . . . , HΩ,κr signiﬁes the spectrum of note r. Wr,t ≥0 denotes the power of note r at time t. As the assumptions(A1) and (A2) do not always hold exactly in reality, an
actual music spectrogram Yω,t may diverge from the “ideal
model” Xω,t to some extent. One way to simplify this kindof deviation process is to assume a probability distribution

power density

The number of consecutive Gaussians with non-zero weights
corresponds to the duration of the note, which we hope toinfer automatically from an observed spectrogram. To thisend, we choose to use a stick-breaking representation [9]to describe the generative process of ur,1 , . . . , ur,M :Vr,m ∼ Beta(Vr,m ; 1, βrV )ur,m = Vr,m

time

m−1
∏

(1 − Vr,m′ ),

(10)
(11)

m′ =1

Figure 3. Power envelope Wr,t of musical note r.
on Yω,t with the expected value of Xω,t . Here, we assume
that Yω,t follows a Poisson distribution with mean Xω,t :Yω,t ∼ Poisson(Yω,t ; Xω,t ),(5)where Poisson(y; x) = xy e−x /y!. It should be noted thatthe maximization of the Poisson likelihood with respect toXω,t amounts to optimally ﬁtting Xω,t to Yω,t by using
the I-divergence as the ﬁtting criterion [3, 8]. To avoid any
indeterminacy in the scaling of Hω,κr and Wr,t , we assume∑
Hω,k = 1 (k = 1, . . . , K).
(6)

which contributes to sparsifying the Gaussian weights.
Now, recall that the onset position Sr (in ticks) of noter is assumed to have been generated via the generative
process described in 2.3. The onset position τr of note r
should thus be placed near the absolute time into which Sr
is converted. Recall also that ψd , which can be considereda function that takes a relative time d as an input and returns the corresponding absolute time as an output, is also
assumed to have been generated (via the generative processdescribed in 2.2). Given Sr and ψd , we ﬁnd it convenient
to write the generative process of τr as
τr ∼ N (τr ; ψSr , (σ τ )2 ).

(12)

2.5 Expansion of generative processωEach spectral proﬁle Hω,k must have the harmonic structure of a particular pitch. One way of ensuring this is to
assume a prior distribution over Hω,k so that it is likely togenerate a spectrum with a certain harmonic structure of
the k-th pitch. Here, we choose to place a Gamma priorover Hω,k , namely

We can describe an expanded version of the generative process of Yω,t as

Hω,k ∼ Gamma(Hω,k ; γ H̄ω,k + 1, β),by introducing an auxiliary variable Cr,m,ω,t . For convenience of analysis, we use this generative process insteadof (5) in the following. Note that it can be readily veriﬁedthat marginalizing out Cr,m,ω,t reduces (13) to (5).

(7)

where Gamma(x; a, b) ∝ xa−1 e−bx . The mode of thisprior distribution is given by H̄ω,k , which should be deﬁned such that it corresponds to the most likely spectral
proﬁle for the k-th pitch. β determines the peakiness ofthe density around the mode.
To incorporate assumption (A3) into Wr,t , we propose
describing Wr,t using a parametric model expressed as asum of Gaussians [8] (Fig. 3):
Wr,t =

M∑

Gr,m,t ,(8)

m=1

wr ur,m −(t−(m−1)φ−τr )2 /2φ2e,Gr,m,t = √2πφwhere wr is the total energy of note r, and τr is the centerof the ﬁrst Gaussian, which can be considered the onset
time of note r (in seconds). The centers of the Gaussians
are constrained so that they are equally spaced with the
distance φ, which is equal to the “standard deviation” of
all the Gaussians. ur,1 , . . . , ur,M are weights associatedwith the M Gaussians, which determine the overall shapeof the power envelope. To avoid any indeterminacy in thescaling of wr and ur,m , we assume
∀r :

M
∑
m=1

ur,m = 1.

(9)

Cr,m,ω,t ∼ Poisson(Cr,m,ω,t ; Hω,κr Gr,m,ω,t )∑()
Yω,t ∼ δ Yω,t −Cr,m,ω,t ,
(13)
r,m

3. APPROXIMATE POSTERIOR INFERENCE
3.1 Variational Bayesian approach
In this section, we describe an approximate posterior inference algorithm for our generative model based on variational inference. The random variables of interest in ourmodel areH = {Hω,k }ω,k : spectrum of pitch k,
w = {wr }r: total energy of note r,
V = {Vr,m }r,m : shape of power envelope of note r,
τ = {τr }r: onset time (sec) of note r,
κ = {κr }r: pitch index assigned to note r,
ψ = {ψd }d: absolute time corresponding to d ticks,µ = {µd }d: local tempo between d and d + 1 ticks,
S = {Sr }r: onset position of note r (in ticks),
L = {Lr }r: duration of note r (in ticks), andϕB , ϕT , ϕN , ϕK : rule probabilities,
which we denote as Θ. Our goal is to compute the posterior p(Θ, C|Y ) where Y = {Yω,t } and C = {Cr,m,ω,t }are sets consisting of observed magnitude spectra and auxiliary variables, respectively. By using the conditional distributions deﬁned in 2.2, 2.3, 2.4, and 2.5, we can write theSonate Opus KV 331 - Tema
Wolfgang Amadeus Mozart  Andante grazioso                  86    






p






B
K
T
N
p(Y, H, w, V, τ, κ, ψ, µ, S, L, ϕ , ϕ , ϕ , ϕ , C)                  6     
  
= p(Y |C)p(C|H, w, V, τ, κ)p(H)p(V )p(w)

8 p

K
K
p(τ |ψ, S)p(ψ|µ)p(µ)p(κ|ϕ )p(ϕ )

score

5
          (a) Correct             

B
B
T
N
T
N



p(S, L|ϕ , ϕ , ϕ )p(ϕ )p(ϕ )p(ϕ ), (14) sf p  





 72              






     but to obtain the exact posterior p(Θ, C|Y ), we need to  

  


compute p(Y ), which involves many intractable integrals.
sf p
60
We can express this posterior variationally as the solu- 10           S  S   S   S         tion to an optimization problem:

 p
sf
sf
sf


48
3

1     2    4      Bar Number  argmin KL(q(Θ, C)∥p(Θ, C|Y )),(15)  q∈Q
sf along with
sf the estimate pof Wr,t(b) Detected beatsflocations
where KL(·∥·) denotes the Kullback-Leibler (KL) diver- 14                             gence between its two arguments. Indeed, if we let Q be

sf p

f



the family of all distributions over Θ and C, the solution to        
   method   with  transcribed
the optimization problem is the exact posterior p(Θ, C|Y ),        (c) Score    the proposed

  



 p 


since KL divergence is minimized exactly when its two
sff
Figure 4. Transcription result obtainedwith the proposed
TEMA

Note Number

joint distribution p(Y, Θ, C) asarguments are equal. Of course, solving this optimization
problem is just as intractable as directly computing the posterior. Although it may appear that no progress has been
made, having a variational formulation allows us to consider tractable choices of Q in order to obtain principledapproximate solutions.
For our model, we deﬁne the set of approximate distributions Q as those that factor as follows:{

Q = q : q(C)q(H)q(w)q(V )q(τ, ψ, µ)q(κ)}
q(S, L)q(ϕK )q(ϕB )q(ϕT )q(ϕN ) . (16)We admit that this is a strong assumption. Its validity andhow it affects the parameter inference result must be investigated in the future.
3.2 Coordinate ascent
We now present an algorithm for solving the optimization
problem described in (15) and (16). Unfortunately, the optimization problem is non-convex, and it is intractable toﬁnd the global optimum. However, we can use a simple coordinate ascent algorithm to ﬁnd a local optimum.
The algorithm optimizes one factor in the mean-ﬁeld approximation of the posterior at a time while ﬁxing all theother factors. The mean-ﬁeld update equations for the variational distributions are given in the following form:q(C ω,t ) = Multinomial(C ω,t ; Yω,t , f Cω,t ),(17)H
H
q(Hω,k ) = Gamma(Hω,k ; ξω,k, ζω,k),w w
q(wr ) = Gamma(wr ; ξr , ζr ),V
V
q(Vr,m ) = Beta(Vr,m ; ξr,m
, ζr,m
),
χ
χ

(18)

q(τ , ψ, µ) = N (χ; ξ , ζ ),q(κr ) = Discrete(κr ; f κr ),

(19)
(20)
(21)
(22)

K
K
q(ϕK
r ) = Dirichlet(ϕr ; ξ r ),(23)

q(Sr , Lr ) = Discrete(Sr , Lr ; f SL
r ),
B
B
B
q(ϕl ) = Dirichlet(ϕl ; ξ l ),
T
T
T
T

(24)

q(ϕ ) = Beta(ϕ ; ξ , ζ ),

(25)
(26)

method applied to Morzart: Piano Sonata No. 11 in A major, K. 331/300i under the situation where τ1 , . . . , τR are
given. In (b), the red and green lines indicate the estimatesof bar lines and the positions of beat locations obtainedwith the present method, respectively.
Public Domain
Sheet music from www.MutopiaProject.org • Free to download,N
N
Nwith theNfreedom to distribute, modify and perform.
Typeset using www.LilyPond .org by Stelios Samelis. Reference: Mutopia-2005/10/29-614

q(ϕ ) = Beta(ϕ ; ξ , ζ ),(27)

This sheet music has been placed in the public domain by the typesetter, for details see: http://creativecommons.org/licenses/publicdomain

where
 
 τ
 τ
η
ν
τ
χ = ψ , ξ χ = η ψ , ζ χ = ν τ ψ
µ
ηµ
ντµ

ντψ
νψ
ν ψµ


ντµ
ν ψµ .
νµ(25)–(27) are performed only when we want to learn therule probabilities. (24)–(27) can be updated using the insideoutside algorithm. The update formulas of the variationalparameters are all given in analytical form, but they areomitted here owing to space limitations.
4. EXPERIMENTAL RESULTS
We now present experimental results obtained with our
proposed model. We ﬁrst conducted a preliminary experiment to conﬁrm that our model can transcribe a score (appropriately estimate the note values of musical notes, beatlocations, and the tempo of a music piece) when the onset times of all the musical notes (namely, τr ’s) are given.
We then show an example of transcription results obtainedusing the complete model directly from an audio spectrogram.
For the ﬁrst experiment, we used a few piano recordings (RWC-MDB-C-2001 No. 26, 27, 30) excerpted fromthe RWC music database [12]. The data were the ﬁrst 10 s,
mixed down to a monaural signal and resampled to 16 kHz.
The constant-Q transform was used to compute spectrograms where the time resolution, the lower bound of thefrequency range, and the frequency resolution were set at
16 ms, 30 Hz and 12 cents, respectively. In this experiment, all the values τ1 , . . . , τR were given manually. The
hyperparameters and initial parameters were set at K =H
H
H74, M = 40, φ = 3, αω,k= βω,kH̄ω,k +1, βω,k= 500, αrw =(a) Detected beat locations along with the estimate of Wr,t

we modeled the generative process of an entire music spectrogram by combining the sub-process by which a musically natural tempo curve is generated, the sub-process bywhich a set of note onset positions is generated based on
a 2-dimensional tree structure representation of music, andthe sub-process by which a music spectrogram is generated according to the tempo curve and the note onset positions. Thanks to this combined generative model, thepresent method performs note extraction and structure estimation simultaneously and thus an optimal solution is obtained within a uniﬁed framework. We described some of
the transcription results obtained with the present method.
6. REFERENCES(b) Score transcribed with the proposed method

Figure 5. Transcription result obtained with the proposed
method applied to Morzart: Piano Sonata No. 11 in A major, K. 331/300i. In (a), the red and green lines indicate
the estimates of bar lines and the positions of beat locations obtained with the present method, respectively. In(b), the red, green and blue circles indicate the deletion
errors, pitch errors and octave errors, respectively.
∑′V= 10e−m/8 / m′ e−m /8 , σ τ = 2, σ ψ =βrw = 0, βr,m1, σ µ = 0.5, αr,k = 2, β T = 1, β N = 2. The initial valuesof Hω,k and H̄ω,k were set at the value obtained with the
non-netaive matrix factorization [13] applied to the magnitude spectrogram of the piano excerpts from the RWC
musical instrument sound database [11]. We set the resolution of the relative time at 4 ticks per quarter note. D
and the initial values of ψd were set at the values obtainedwith [10]. The algorithm was run for 10 iterations. After
convergence, we took the expected values of the posteriors
and regarded them as the parameter estimates.
Fig. 4 shows an example of the score we obtained when
we applied the present method to Mozart’s Sonata (RWCMDB-C-2001 No. 26). As can be seen from this example,
the note values and the beat locations were appropriately
estimated. We also conﬁrmed that reasonably good results
were obtained for other recordings such as Chopin’s Nocturne No. 2 in E♭-maj, Op. 9 (RWC-MDB-C-2001 No. 30).
For the second experiment, we applied our method without providing any information about τ . The experimental
conditions were the same as above except that we assumed
that τ was unknown. Fig. 5 shows an example of the estimates of Wr,t (namely, the power envelope of note r) andthe score obtained with the present method applied to thesame data in Fig. 5. The result showed that many octave
errors had occurred. This kind of error often occurs when
there is a mismatch between a spectrum model and an actual spectrum. The validity of the assumptions we havemade about the spectra of musical sounds in 2.4 must be
carefully examined in the future.
5. CONCLUSION
This paper proposed a Bayesian model for automatic music transcription. Automatic music transcription involves
several interdependent subproblems: multiple fundamental
frequency estimation, onset detection, and rhythm/tempo
recognition. To circumvent the chicken-and-egg problem,[1] N. Bertin, R. Badeau, and G. Richard, “Blind signal decompositions for automatic transcription of polyphonicmusic: NMF and K-SVD on the benchmark,” In Proc.
ICASSP2007, Vol. 1, pp. 65–68, 2007.
[2] K. Ochiai, H. Kameoka, and S. Sagayama, “Explicit beat structure modeling for non-negative matrix factorization-based multipitch analysis,” in Proc.
ICASSP2012, pp. 133–136, 2012.
[3] A. T. Cemgil, “Bayesian inference in non-negative matrix factorisation models,” Technical Report CUED/FINFENG/TR.609, University of Cambridge, 2008.
[4] M. D. Hoffman, D. M. Blei, and P. R. Cook, “Bayesiannonparametric matrix factorization for recorded music,” in Proc. ICML2010, pp. 439–446.
[5] K. Yoshii, and M. Goto, “A nonparametric Bayesian
multipitch analyzer based on inﬁnite latent harmonic
allocation,” IEEE Trans. Audio, Speech, Language
Process., Vol. 20, No. 3, pp. 717–730, 2012.
[6] M. Nakano, Y. Ohishi, H. Kameoka, R. Mukai, and
K. Kashino, “Bayesian nonparametric music parser,”in Proc. ICASSP2012, pp. 461–464, 2012.
[7] P. Liang, S. Petrov, M. I. Jordan, and D. Klein, “Theinﬁnite PCFG using hierarchical Dirichlet processes,”in EMNLP2007, pp. 688–697.
[8] H. Kameoka, T. Nishimoto, and S. Sagayama, “A multipitch analyzer based on harmonic temporal structured
clustering,” IEEE Trans. on Audio, Speech, Language
Process., Vol. 15, No. 3, pp. 982–994, 2007.
[9] J. Sethuraman, “A constructive deﬁnition of Dirichletpriors,” Statistica Sinica, Vol. 4, pp. 639–650, 1994.
[10] D. P. W. Ellis, “Beat tracking by dynamic programming,” Journal of New Music Research, 36(1):51–60,
2007.
[11] M. Goto, “Development of the RWC music database,”
In Proc. the 18th International Congress on Acoustics(ICA 2004), pp. I-553–I-556, 2004.
[12] M Goto, H Hashiguchi, T Nishimura, and R Oka,“RWC music database: Popular, classical, and jazzmusic database,” In Proc. ISMIR, pp. 287–288, 2002.
[13] P. Smaragdis and J. C. Brown, “Non-negative matrix
factorization for music transcription,” in Proc. WASPAA2003, pp. 177–180, 2003.