Context-free 2D tree structure model of musical notes forBayesian modeling of polyphonic spectrograms
1,2
1
1
Hirokazu Kameoka , Kazuki Ochiai , Masahiro Nakano ,1
1
Masato Tsuchiya , Shigeki Sagayama

1 The University of Tokyo, 2 NTT Communication Science Laboratories

 Note value estimation
 Tempo/beat estimation

 Ambiguities involved inthese subproblems:

TimeRhythmanalysis

Multipitch analysis

For each node n in the parse tree:

Spectrogram of mixtureof two notes (C3 and G3)[Choose either “EMISSION” or “BINARY-PRODUCTION”]
OverlappingcomponentC3TimeIfNote number

Power of note r
at time t
Spectrum of note r[A3] We describe powerenvelope
using aparametric function:

OnsetTimeChild 2→→

Parent andchildren nodes= TIME-SPANNING

Onset positionof note r should be placed near
the absolute time into whichis converted. To
ensure this, we assume.
Finally, we assume that observed
spectrogram is generated
according to:

TimeNote: The maximization of the Poisson likelihood withrespect toamounts to optimally fitting
toby using the I-divergence as the fitting criterion.

Split note n into
two consecutive
notes n1 and n2...

Time= SYNCHRONIZATION

If

Spectrogram can be described asTime-spanproduction→

Produce 2 copies
of note nPossible interpretations

Rhythm analysis[A1][A2]→= BINARY-PRODUCTIONIf

be multiple interpretations of which notes are present
in a mixture.

Timewith pitch index[A1] Each musical note has static spectral
1, 2, 3
profile characterized by particular pitch.
[A2] Magnitude spectrum is represented bysuperposition of spectra of multiple notes.
[A3] Power of each musical note varies
continuously in time.

[Choose either “SYNCHRONIZATION” or “TIME-SPANNING”]

 Since each note consists of many overtones, there canChild 1Emit terminal symbol

Log-frequency

G3ParentTick

We make following assumptions aboutsource spectra:
Spectrum templates

Time-spanning binary
production rules= EMISSION

If



...

Log-frequency

: Onset position (in units
Synchronizationof ticks) of note r
 Music has a 2D hierarchicalstructure, consisting ofTime-spanning structureSynchronizing structure We introduce extension of probabilistic context-free grammar (PCFG):

Note number

Rhythm analysis

TickTime-spanproduction

tick

Subprocess ①

Log-frequency

 Onset/offset estimation
Time

Multipitchanalysis

note onset positions*according

Pitch #Amplitude

 Multiple F0 estimation② Subprocess for generating

For each note r, we assign pitch index...
to*Subprocess ②

can be regarded
as the overall tempo curve of aTime
piece of music
 To ensure smooth overall change in tempo, we assume:

subproblems that are interdependent ofeach other: Audio source separation

: time)

Pitch #

Tick : relative time notion (integer)(the number of discrete divisions a quarter note has been split into) Instantaneous (or local) tempo: the real duration (in units ofseconds) corresponding to the interval between d and d + 1 ticks
: the absolute time (in units of seconds) corresponding to ticks( : frequency, Automatic music transcription consists ofMultipitch analysis

If we consider 16 divisions per quarter
note, the duration of 40 ticks
corresponds to two-and-a-half beats.

...

① Subprocess for generating tempo curve

1. Introduction

③ Subprocess for generating music spectrogramPower

automatic music transcription. Automatic musictranscription involves several subproblems that areinterdependent of each other: multiple fundamental
frequency estimation, onset detection, and rhythm /tempo recognition. We formulate a unified generativemodel, which could be used to jointly solve the problemsof determining the pitch and onset time of each musicalnote, the rhythm and the overall tempo variation of apiece of music, with a Bayesian approach.

3. Generative Model

Pitch #Abstract: This paper proposes a Bayesian model for Since we cannot define a note value without having anotion for tempo and vice versa, there can be infinite
interpretations regarding what the intended rhythm
was and how the tempo varied.

 “Chicken-and-egg” interdependencies

4. Posterior Inference

5. Music Transcription Experiments

 Variational inference Experiment 1

Goal is to compute

Audio source
separation

Multiple F0
estimation
Onset/offset
estimation
Tempo/beatestimation

Note value
estimation

 Simultaneous estimation is generally preferable whenseveral estimation problems are interdependent!

2. Motivation

To obtain exact posterior, we must derive
marginal distribution, but this involves
intractable integrals.

 How do humans recognize music ?
Humans can recognize music by combining
 hypothesis made according tothe observation about when
and which notes are active, andLog-frequency

common sense about how music and music
performance should be (e.g., how onset
occurrences of notes should be temporallystructured, how tempo varies in time)
NaturalNatural

 Experiment 2Next, we applied ourmethod to the same datawithout providing any
information about .
 Result: (d) and (e)(d) Detected beat locations along with the estimate of Wr,t

Kullback-Leibler divergence
betweenandTime

UnnaturalTime

Time

Tempo

Instead, we consider finding

Tractable choices ofallows us toobtain principled approximate solutions
Mean-field approximation



Coordinate ascent

NaturalTempo

Tempo



Test data: piano recording(RWC-MDB-C-2001 No. 26)
Morzart: Piano Sonata
No. 11 in A major,(b) Detected beat locations along with the estimate of Wr,tK. 331/300i
 Monaural
sampling rate=16kHz
 Constant-Q transform wasused to obtain the observedspectrogram.
 Note onsets
were(c) Score transcribed with the proposed method
given manually.
 Result: (b) and (c): total energy of note r: shape of power envelope of note r: onset time (sec) of note r: pitch index assigned to note r: absolute time corresponding to d ticks: local tempo between d and d+1 ticks: onset position of note r (in ticks): duration of note r (in ticks)

Rhythm(a) Correct score

UnnaturalNaturalTime We propose to model the generative process ofan entire spectrogram of a piece of music byformulating the following three sub-processes
and combining them into one process:①subprocess by which a tempo curve is generated,②subprocess by which a set of note onset positions(in terms of the relative time) is generated, and③subprocess by which spectrogram is generated

Parameter inference given the observation would thengive a musically likely interpretation of what is actually
happening in the spectrogram (i.e., a musical score).

 By iteratively optimizing one factor q(.) in the mean-fieldapproximation of the posterior at a time while fixing all theother factors, we can find a local optimum.

Probability distributions(e) Score transcribed with the proposed method

Referenceswherewhereand

The result showed that many octave
errors had occurred. This kind of error
often occurs when there is a mismatch
between a spectrum model and an
actual spectrum. The validity of theassumptions we have made about
the spectra of musical sounds must
be carefully examined in the future.

andwith[1] N. Bertin, R. Badeau, and G. Richard, “Blind signal decompositions for automatic transcription of polyphonicmusic: NMF and K-SVD on the benchmark,” in Proc. ICASSP, Vol. 1, pp. 65-68, 2007.
[2] K. Ochiai, H. Kameoka, and S. Sagayama, “Explicit beat structure modeling for non-negative matrix
factorization-based multipitch analysis,” in Proc. ICASSP, pp. 133-136, 2012.
[3] A. T. Cemgil, “Bayesian inference in non-negative matrix factorisation models,” Technical Report CUED/FINFENG/TR.609, University of Cambridge, 2008.
[4] M. D. Hoffman, D. M. Blei, and P. R. Cook, “Bayesian nonparametric matrix factorization for recorded music,”in Proc. ICML, pp. 439-446, 2010.
[5] K. Yoshii, and M. Goto, “A nonparametric Bayesian multipitch analyzer based on infinite latent harmonicallocation,” IEEE Trans. Audio, Speech, Language Process., Vol. 20, No. 3, pp. 717-730, 2012.
[6] M. Nakano, Y. Ohishi, H. Kameoka, R. Mukai, and K. Kashino, “Bayesian nonparametric music parser,” inProc. ICASSP, pp. 461-464, 2012.
[7] P. Liang, S. Petrov, M. I. Jordan, and D. Klein, “The infinite PCFG using hierarchical Dirichlet processes,” inProc. EMNLP, pp. 688-697, 2007.
[8] H. Kameoka, T. Nishimoto, and S. Sagayama, “A multipitch analyzer based on harmonic temporal structuredclustering,” IEEE Trans. on Audio, Speech, Language Process., Vol. 15, No. 3, pp. 982-994, 2007.