A HIERARCHICAL BAYESIAN MODEL OF CHORDS, PITCHES, AND
SPECTROGRAMS FOR MULTIPITCH ANALYSIS
Yuta Ojima1

1

Eita Nakamura1
Katsutoshi Itoyama1
Kazuyoshi Yoshii1
Graduate School of Informatics, Kyoto University , Japan{ojima, enakamura}@sap.ist.i.kyoto-u.ac.jp, {itoyama, yoshii}@kuis.kyoto-u.ac.jp

ABSTRACTThis paper presents a statistical multipitch analyzer thatcan simultaneously estimate pitches and chords (typical
pitch combinations) from music audio signals in an unsupervised manner. A popular approach to multipitch analysis is to perform nonnegative matrix factorization (NMF)
for estimating the temporal activations of semitone-level
pitches and then execute thresholding for making a pianoroll representation. The major problems of this cascading
approach are that an optimal threshold is hard to determine
for each musical piece and that musically inappropriate
pitch combinations are allowed to appear. To solve these
problems, we propose a probabilistic generative model thatfuses an acoustic model (NMF) for a music spectrogramwith a language model (hidden Markov model; HMM) for
pitch locations in a hierarchical Bayesian manner. Morespecifically, binary variables indicating the existences ofpitches are introduced into the framework of NMF. The latent grammatical structures of those variables are regulatedby an HMM that encodes chord progressions and pitch cooccurrences (chord components). Given a music spectrogram, all the latent variables (pitches and chords) are estimated jointly by using Gibbs sampling. The experimental
results showed the great potential of the proposed method
for unified music transcription and grammar induction.
1. INTRODUCTION
The goal of automatic music transcription is to estimate thepitches, onsets, and durations of musical notes containedin polyphonic music audio signals. These estimated values
must be directly linked with the elements of music scores.
More specifically, in this paper, a pitch means a discrete
fundamental frequency (F0) quantized in a semitone level,
an onset means a discrete time point quantized on a regulargrid (e.g., eighth-note-level grid), and a duration means a
discrete note value (integer multiple of the grid interval).
In this study we tackle multipitch estimation (subtask of
automatic music transcription) that aims to make a binary
piano-roll representation from a music audio signal, wherec Yuta Ojima, Eita Nakamura, Katsutoshi Itoyama,
Kazuyoshi Yoshii. Licensed under a Creative Commons Attribution 4.0
International License (CC BY 4.0). Attribution: Yuta Ojima, Eita Nakamura, Katsutoshi Itoyama, Kazuyoshi Yoshii. “A Hierarchical BayesianModel of Chords, Pitches, and Spectrograms for Multipitch Analysis”,
17th International Society for Music Information Retrieval Conference,2016.

Language model
E♭

A♭ E♭

F

Chords
Pitches

Bases

Activations

Spectrograms

Acoustic model

Figure 1. Overview of the proposed model consisting of
language and acoustic models that are linked through binary variables S representing the existences of pitches.
only the existences of pitches are estimated at each frame.
A popular approach to this task is to use non-negative matrix factorization (NMF) [1–7]. It approximates the magnitude spectrogram of an observed mixture signal as the
product of a basis matrix (a set of basis spectra corresponding to different pitches) and an activation matrix (a set of
temporal activations corresponding to those pitches). The
existence of each pitch is then determined by executing
thresholding or Viterbi decoding based a hidden Markov
model (HMM) for the estimated activations [7, 8].
This NMF-based cascading approach, however, has two
major problems. First, it is hard to optimize a threshold foreach musical piece. Second, the estimated results are allowed to be musically inappropriate because the relationships between different pitches are not taken into account.
In fact, music has simultaneous and temporal structures;certain kinds of pitches (e.g., C, G, and E) tend to simultaneously occur to form chords (e.g., C major), which varyover time to form typical progressions. If such structural
information is unavailable for multipitch analysis, we needto tackle the chicken-and-egg problem that chords are determined by pitch combinations, and vice versa.
To solve these problems, we propose a statistical methodthat can discover chords and pitches from music audio signals in an unsupervised manner while taking into accounttheir interdependence (Fig.1). More specifically, we formulate a hierarchical Bayesian model that represents thegenerative process of an observed music spectrogram by
unifying an acoustic model (probabilistic model underlying NMF) that represents how the spectrogram is generated
from pitches and a language model (HMM) that represents
how the pitches are generated from chords. A key feature of the unified model is that binary variables indicating
the existences of pitches are introduced into the frameworkof NMF. This enables the HMM to represent both chord

309

310

Proceedings of the 17th ISMIR Conference, New York City, USA, August 7-11, 2016transitions and pitch combinations using only discrete variables forming a piano-roll representation with chord labels.
Given a music spectrogram, all the latent variables (pitches
and chords) are estimated jointly by using Gibbs sampling.
The major contribution of this study is to realize unsupervised induction of musical grammars from music audio
signals by unifying acoustic and language models. This approach is formally similar to, but essentially different fromthat to automatic speech recognition (ASR) because boththe models are jointly learned in an unsupervised manner.
In addition, our unified model has a three-level hierarchy(chord–pitch–spectrogram) while ASR is usually based ona two-level hierarchy (word–spectrogram). The additional
layer is introduced by using an HMM instead of a Markov
model (n-gram model) as a language model.
2. RELATED WORKThis section reviews related work on multipitch estimation(acoustic modeling) and on music theory implementationand musical grammar induction (language modeling).
2.1 Acoustic Modeling
The major approach to music signal analysi is to use nonnegative matrix factorization (NMF) [1–6, 9]. Cemgil etal. [9] developed a Bayesian inference scheme for NMF,
which enabled the introduction of various hierarchical priorstructures. Hoffman et al. [3] proposed a Bayesian nonparametric extension of NMF called gamma process NMF
for estimating the number of bases. Liang et al. [6] proposed beta process NMF, in which binary variables are introduced to indicate the needs of individual bases at eachframe. Another extension is source-filter NMF [4], whichfurther decomposes the bases into sources (corresponding
to pitches) and filters (corresponding to timbres).
2.2 Language Modeling
The implementation and estimation of music theory behindmusical pieces are composed have been studied [10–12].
For example, some attempts have been made to computationally formulate the Generative Theory of Tonal Music (GTTM) [13], which represents the multiple aspects ofmusic in a single framework. Hamanaka et al. [10] reformalized GTTM through a computational implementation and developed a method for automatically estimating
a tree that represents the structure of music, called a timespan tree. Nakamura et al. [11] also re-formalized GTTM
using a probabilistic context-free grammar model and proposed inference algorithms. These methods enabled automatic analysis of music. On the other hand, inductionof music theory in an unsupervised manner has also beenstudied. Hu et al. [12] extended latent Dirichlet allocationand proposed a method for determining the key of a musical piece from symbolic and audio music based on thefact that the likelihood of appearance of each note tends
to be similar among musical pieces in the same key. Thismethod enabled the distribution of notes in a certain key tobe obtained without using labeled training data.

Assuming that the concept of chords is a kind of music
grammar, statistical methods of supervised chord recognition [14–17] are deeply related with unsupervised musical grammar induction. Rocher et al. [14] attempted chord
recognition from symbolic music by constructing a directed
graph of possible chords and then calculating the optimal
path. Sheh et al. [15] used acoustic features called chroma
vectors to estimate chords from music audio signals. They
constructed an HMM whose latent variables are chord labels and whose observations are chroma vectors. Maruo
et al. [16] proposed a method that uses NMF for extracting reliable chroma features. Since these methods needlabeled training data, the concept of chords is required inadvance. Approaches to make use of a sequence of chordsin estimating pitches has also been proposed [18, 19]. Thismethod estimates chord progressions and multiple pitches
simultaneously by using a dynamic Bayesian network andshows better performance even with a simple acoustic model.
Recent works employ recurrent neural networks as a language model to describe the relations between pitch combinations [20, 21].
3. PROPOSED METHODThis section explains the proposed method of multipitch
analysis that simultaneously estimates pitches and chordsat the frame level from music audio signals. Our approachis to formulate a probabilistic generative model for observed music spectrograms and then solve the “inverse”
problem, i.e., given a music spectrogram, estimate unknown
random variables involved in the model. The proposedmodel has a hierarchical structure consisting of acoustic
and language models that are connected through a pianoroll, i.e., a set of binary variables indicating the existencesof pitches (Fig. 1). The acoustic model represents the generative process of a music spectrogram from the piano roll,basis spectra, and temporal activations of individual pitches.
The language model represents the generative process ofchord progressions and pitch locations from chords.
3.1 Problem Specification
The goal of multipitch estimation is to make a piano roll×T
from a music audio signal. Let X ∈ RF
be the mag+nitude spectrogram of a target signal, where F is the number of frequency bins and T is the number of time frames.
We aim to convert X into a piano roll S ∈ {0, 1}K×T ,which represents the existences of K kinds of pitches overT frames. In addition, we attempt to estimate a sequenceof chords Z = {zt }Tt=1 .
3.2 Acoustic Modeling
The acoustic model is formulated in a similar way to betaprocess NMF having binary masks [6] (Fig. 2). The given
×T
spectrogram X ∈ RF
is factorized into bases W ∈+F ×K
R+ , activations H ∈ RK×T
, and binary variables S ∈+{0, 1}K×T as follows:PK
Xf t |W , H, S ∼ Poisson(1)
k=1 Wf k Hkt Skt ,Proceedings of the 17th ISMIR Conference, New York City, USA, August 7-11, 2016

311
Chord progression

E♭

Activations

A♭

E♭

F

B♭

Hadamard product

Binary variables

follows transition probabilities

Bases

Binary variables
Corresponding toa piano-roll representation

follows emission probabilities
84 pitches

Spectrograms

Figure 2. The overview of the acoustic model based on avariant of NMF having binary variables (masks).

Figure 3. The overview of the language model based on
an HMM that stochastically emits binary variables.

where {Wf k }F
f =1 is the k-th basis spectrum, Hkt is thevolume of basis k at frame t, and Skt is a binary variableindicating whether or not basis k is used at frame t.
A set of basis spectra W is divided into two parts: harmonic spectra and noise spectra. In this study we prepare
Kh harmonic basis spectra corresponding to Kh different
pitches and one noise basis spectrum (K = Kh + 1). Assuming that the harmonic structures of the same instrument
have the shift-invariant relationships, the harmonic part of
W are given byh F{Wf k }F(2)
f =1 = shift {Wf }f =1 , ζ(k − 1) ,3.3 Language Modelingfor k = 1, . . . Kh , where {Wfh }F
f =1 is a harmonic template
structure common to harmonic basis spectra used for NMF,
shift (x, a) is an operator that shifts x = [x1 , . . . , xn ]Tto [0, . . . , 0, x1 , . . . , xn−a ]T , and ζ is the number of frequency bins corresponding to the semitone interval.
We put two kinds of priors on the harmonic templaten F
spectrum {Wfh }F
f =1 and a noise basis spectrum {Wf }f =1 .
To make the harmonic spectrum sparse, we put a gamma
prior on {Wfh }F
f =1 as follows:
Wfh ∼ G ah , bh(3)where ah and bh are hyperparameters. On the other hand,
we put an inverse-gamma chain prior [22] on {Wfn }F
f =1 toinduce the spectral smoothness as follows:


n
W
ηW
GW
f |Wf −1 ∼ IG η , Wf −1 ,


W ηW
,
Wfn |GW
f ∼ IG η , GW

(4)f

where η W is a hyperparameter that determines the strengthof smoothness and GW
f is an auxiliary variable that induces
positive correlation between Wfn−1 and Wfn .
A set of activations H is represented in the same way
as W . If Hkt takes almost zero, Skt has no impact on
NMF. This allows Skt to take one (the corresponding pitch
is judged to be activated) even though the activation Hkt
is almost zero. We can avoid this problem by putting an
inverse-gamma prior for Hkt to induce non-zero values.
To induce the temporal smoothness in addition, we put thefollowing inverse-gamma chain prior on H:


ηH
GH
,
kt |Hk(t−1) ∼ IG ηH , Hk(t−1)

H
Hkt |Gkt ∼ IG ηH , GηH
,
H

(5)

kt

where ηH is a hyperparameter that determines the strengthof smoothness and GH
kt is an auxiliary variable that induces
positive correlation between Hk(t−1) and Hkt .

The language model is an HMM that has a Markov chain oflatent variables Z = {z1 , . . . , zT } (zt ∈ {1, . . . , I}) andemits binary variables S = {s1 , . . . , sT } (st ∈ {0, 1}Kh ),where I represents the number of states (chords) and Kh
represents the number of possible pitches. Note that S isactually a set of latent variables in the proposed unified
model. The HMM is defined as:z1 |φ ∼ Categorial(φ),

zt |zt−1 , ψ zt−1 ∼ Categorical(ψ zt−1 ),Skt |zt , πzt k ∼ Bernoulli(πzt k )

(6)
(7)
(8)

where ψ i ∈ RI is a set of transition probabilities of chordi, φ ∈ RI is a set of initial probabilities, and πzt k indicatesthe probability that the k-th pitch is emitted under a chord
zt , We put conjugate priors on these parameters as:ψ i ∼ Dir(1I ),φ∼ Dir(1I ),πzt k ∼ Beta(e, f ),(9)where 1I is the I-dimensional all-one vector and e and f
are hyperparameters.
In practice, we represent only the emission probabilities of 12 pitch classes (C, C#, . . ., B) in one octave. Those
probabilities are copied and pasted to recover the emission
probabilities of Kh kinds of pitches. In addition, the emishsion probabilities {πik }K
k=1 of chord i are forced to havecircular-shifting relationships with those of other chords ofthe same type. In this paper, we consider only major andminor chords as chord types (I = 2 × 12) for simplicity.
3.4 Posterior Inference
Given the observed data X, our goal is to calculate theposterior distribution p(W , H, S, z, π, ψ|X). Since analytic calculation is intractable, we use Markov chain Monte
Carlo (MCMC) methods as in [23]. Since the acoustic
and language models share only the binary variables, eachmodel can be updated independently when the binary variables are given. These models and binary variables are iteratively sampled. Finally, the latent variables (chord progressions) of the language model are estimated by using
the Viterbi algorithm and the binary variables (pitch locations) are determined by using parameters having the maximum likelihood.
3.4.1 Sampling Binary Variables
The binary variables S are sampled from a posterior distribution that is calculated by integrating the acoustic model312

Proceedings of the 17th ISMIR Conference, New York City, USA, August 7-11, 2016as a likelihood function and the language model as a priordistribution according the Bayes’ rule. Note that as shownin Fig. 1, the binary variables S are involved in both acoustic and language models (i.e., the probability of each pitch
being used is determined by a chord, and whether or noteach pitch is used affects the reconstructed spectrogram).
The conditional posterior distribution of Skt is given by1Skt ∼ Bernoulli P1P+P,(10)
0
where P1 and P0 are given by

P1 = p(Skt = 1|S¬k,t , xt , W , H, π, z, α)
(11)


X
ft
Q
∝ πzαk f X̂f¬kexp{−Wf k Hkt },t + Wf k Hkt

P0 = p(Skt = 0|S¬k,t , xt , W , H, π, α)
Xf t
Q 
∝ (1 − πzk )α f X̂f¬k
,
(12)
t
P
where X̂f¬kt ≡l6=k Wf l Hlt Slt denotes the magnitudeat frame t reconstructed without using the k-th basis andα is a parameter that determines the weight of the language model relative to that of the acoustic model. Such aweighting factor is also needed in ASR. If α is not equal toone, Gibbs sampling cannot be used because the normalization factor cannot be analytically calculated. Instead,the Metropolis-Hastings (MH) algorithm is used by regarding Eq. (10) is used as a proposal distribution
3.4.2 Updating the Acoustic Model
The parameters of the acoustic model W h , W n , and H
can be sampled using Gibbs sampling. These parameters
are categorized into those having gamma priors (W h ) andthose inverse-gamma chain priors (W n and H).
Using the Bayes’ rule, the conditional posterior distribution of W h is given by

P
P
h
h
Wfhk ∼ G, (13)t Xf t λf tk + a ,t Hkt Skt + b
where λf tk is a normalized auxiliary variable that is calculated with the latest sampled variables Ŵ , Ĥ, and Ŝ,
as:
Ŵ

Hˆ Sˆ

(14)

kt
λf tk = P fWkˆ kt
.
Hˆ Sˆ
l

fl

lt

lt

The other parameters are sampled through auxiliary variables. Since H and GH are interdependent in Eq. (5) andcannot be sampled jointly, GH and H are sampled altenately. The conditional posterior of GH is given by



1
1
GH
.
(15)
kt ∼ IG 2ηH , ηH Hkt + Hk(t−1)

Similarly, the conditional posteriors of H, GW , and W n
are given by
Hkt ∼ IG 2ηH , ηH GH 1
+ G1H
,
kt
k(t+1)

1GW+ W n1,f ∼ IG 2ηW , ηW
Wfn
f −1



1
Wfn ∼ IG 2ηW , ηW GW
+ G1W
,
f +1

(16)
(17)
(18)

f

if the observation X is not taken into account. Using theBayes’ rule and Jensen’s inequality as in Eq. (13) and regarding Eq. (16) as a prior, the conditional posterior con-sidering the observation X is written as follows: 1
PP
Hkt ∼ GIG 2Skt f Wf k , δH , f Xf t λf tk − γH ,where γH = 2ηH and δH = ηH ( GH1

k(t+1)+ G1H ). The
kt

conditional posterior of W n can be derived in the samemanner as follows:P
P
Wfnk ∼ GIG (2 t Hkt Skt , δW , t Xf t λf tk − γW ) ,where γW = 2ηW and δW = ηW ( GW1

f +1+ G1W )
f

3.4.3 Updating the Language Model
The latent variables Z are sampled from the following conditional posterior distribution:p(zt |S, π, φ, Ψ) ∝ p(s1 , . . . , st , zt ),(19)

where π is the emission probabilities, φ is the initial probabilities, and Ψ = {ψ 1 , . . . , ψ I } is a set of the transition probabilities from each state. The right-hand side ofEq. (19) is further factorized using the conditional independence over Z and S as follows:p(s1 , . . . , st , zt )P
= p(st |zt ) zt−1 p(s1 , . . . , st−1 , zt−1 )p(zt |zt−1 ), (20)p(s1 , z1 ) = p(z1 )p(s1 |z1 ) = φz1 p(s1 |πz1 ).

(21)

Using Eqs. (20) and (21) recursively, p(s1 , . . . , sT |zT ) canbe efficiently calculated via forward filtering and the lastvariable zT is sampled according to zT ∼ p(s1 , . . . , sT |zT ).
If the latent variables zt+1 , . . . , zT are given, zt is sampledfrom a posterior given byp(zt |S, zt+1 , . . . , zT ) ∝ p(s1 , . . . , st , zt )p(zt+1 |zt ). (22)Since p(s1 , . . . , st , zt ) can be calculated in Eq. (20), zt isrecursively sampled from zt ∼ p(s1 , . . . , st , zt )p(zt+1 |zt )via backward sampling.
The posterior distribution of the emission probabilities
π is given by using the Bayes’ rule as follows:p (π|S, z, φ, Ψ) ∝ p (S|π, z, φ, Ψ) p (π) .

(23)This is analytically calculable because p (π) is a conjugate
prior of p (S|π, z, φ, Ψ). Let Ci be the number
P of occurrences of chord i ∈ {1 . . . I} in Z and ci ≡ t∈{t|zt =i} st
be a K-dimensional vector that denotes the sum of st under the condition zt = i. The parameters π are sampledaccording to a conditional posterior given byπ ∼ Beta (e + cik , f + Ci − cik ) .

(24)

The posterior distributions of the transition probabilities ψ and the initial probabilities π are given similarly as
follows:p(φ|S, z, π, Ψ) ∝ p(z1 |φ) p(φ)(25)
Q
p(ψ|S, z, π, φ) ∝ t p(zt |zt−1 , ψ zt−1 ) p(ψ zt−1 ). (26)Since p(φ) and p (ψ i ) are conjugate priors of p(z1 |φ) andp(zt |zt−1 , ψ zt−1 ), respectively, these posteriors can be easily calculated. Let ei be the unit vector whose i-th elementp

1 GIG(a, b, p) ≡bax+(a/b) 2√
xp−1 exp(− 2 x ) denotes a general2Kp ( ab)ized inverse Gaussian distribution.

Proceedings of the 17th ISMIR Conference, New York City, USA, August 7-11, 2016313

is 1 and ai be the I-dimensional vector whose j-th element
denotes the number of transition from state i to state j. The
parameters φ and ψ i are sampled according to conditional
posteriors given byφ ∼ Dir (1I + ez1 ) ,ψ i ∼ Dir (1I + ai ) .

(27)

4. EVALUATION
We report comparative experiments we conducted to evaluate the performance of our proposal model in pitch estimation. First, we confirmed in a preliminary experimentthat correct chord progressions and emission probabilities
were estimated from the piano-roll by the language model.
Then, we estimated the piano-roll representation from acoustic audio signals by using the hierarchical model and theacoustic model.
4.1 Experimental Conditions
We used 30 pieces (labeled as “ENSTDkCl”) selected from
the MAPS database [24]. We converted them into monaural signals and truncated each of them to 30 seconds fromthe beginning. The magnitude spectrogram was made byusing the variable-Q transform [25]. The 926 × 10075
spectrogram thus obtained was resampled to 926 × 3000
by using MATLAB’s resample function. Moreover, weused harmonic and percussive source separation (HPSS)[26] as a preprocessing. Unlike the original study, HPSS
was performed in the log-frequency domain. Median filter is applied over 50 time frames and 40 frequency bins
each. Hyperparameters were empirically determined as
I = 24, ah = 1, bh = 1, an = 2, bn = 1, c = 2, d = 1, e =5, f = 80, α = 1300, ηW = 800000 and ηH = 15000.
The emission probabilities are obtained for 12 notes, which
are expanded to cover 84 pitches. In practice, we fixed theprobability of internal transition (i.e. p(zt+1 = zt |zt )) toa large value (1 − 8.0 × 10−8 ) and assumed that the probabilities of transition to a different state follow Dirichlet
distribution as shown in section 3.4.3 We implemented the
proposed method by using C++ and a linear algebra librarycalled Eigen3. The estimation was conducted with a standard desktop computer with an Intel Core i7-4770 CPU(8-core, 3.4 GHz) and 8.0 GB of memory. The processing
time for the proposed method with one music piece (30seconds as mentioned above) was 15.5 minutes .
4.2 Chord Estimation for Piano Rolls
We first verified that the language model properly estimated the emission probabilities and a chord progression.
As an input, we combined correct binary piano-roll representations for 84 pitches (MIDI numbers 21–104) of thepieces we used. Since each representation has 3000 timeframes and we used 30 pieces, the input was 84×90000
matrix. We evaluated the precision of chord estimation
as the ratio of the number of frames whose chords were
estimated correctly to the total number of frames. Since
we prepared two chord types for each root note, we treated“major” and “7th” in the ground-truth chords as “major” inthe estimated chords, and “minor” and “minor 7th” in the

Figure 4. Emission probabilities estimated in the preliminary experiment. The left corresponds to major chords andthe right corresponds to minor chords.
ground-truth chords as “minor” in the estimated chords.
In evaluation, other chord types were not used in evaluation and chord labels were estimated to maximize the precision since we estimated chords in an unsupervised manner. Since original MAPS database doesn’t contain chord
information, one of the authors labeled chord informationfor each music piece by hand 2 .
The experimental results shown in Fig. 4 shows that major chords and minor chords, which are typical chord types
in tonal music, were obtained as emission probabilities.
This implies that we can obtain the concept of chord frompiano-roll data without any prior knowledge. The precision was 61.33%, which indicates our model estimates
chords correctly to some extent even in an unsupervisedmanner. On the other hand, other studies on chord estimation have reported higher score [15, 16]. This is becausethat they used labeled training data and that they evaluatedtheir method with popular music, which has clearer chord
structure than classical music we used.
4.3 Multipitch Estimation for Music Audio Signals
We then evaluated our model in terms of the frame-level
recall/precision rates and F-measure:
P

c

P

c

2RP
R = Pt rtt , P = Pt ett , F = R+P
,
t

t

(28)

where rt , et , and ct are respectively the numbers of groundtruth, estimated and correct pitches at the t-th time-frame.
To cope with the arbitrariness in octaves of the obtainedbases, estimated results for the whole piece were shiftedby octaves and the most accurate one was used for theevaluation. We conducted a few comparative experiments
under the following conditions: 1) Chords were fixed andunchanged during a piece (the acoustic model), 2) the language model was pre-trained using the correct chord labels
and a correct piano-roll, and the learned emission probabilities were used in estimation (pre-trained with chord),3) the language model was pre-trained using only a correct piano-roll, and the learned emission probabilities wereused in estimation (pre-trained without chord). we evaluated the performances under the second and the third conditions by using cross-validation.
As shown in Table 1, the performance of the proposed
method in the unsupervised setting (65.0%) was better thanthat of the acoustic model (64.7%). As shown in Fig. 5, theF-measure improvement due to integrating the language
model for each piece correlated positively with the preci2 The annotation data used for evaluation is available on
http://sap.ist.i.kyoto-u.ac.jp/members/ojima/mapschord.zip

314

Proceedings of the 17th ISMIR Conference, New York City, USA, August 7-11, 2016
Condition

F

RP

The integrated model
The acoustic model

65.0
64.7

67.3
64.7

62.8
64.7

Pre-trained w/ chord
Pre-trained w/o chord

65.5
65.0

65.3
65.5

65.6
64.6

Table 1. Experimental results of multipitch analysis for 30
piano pieces labeled as ENSTDkCl.
90

Figure 6. Emission probabilities learned from estimatedpiano-roll. Chord structures like those in Fig. 4 were obtained.

Chord precision [%]

80
70
60
50
40
30
20−4

−3

−2

−1

0

1

2

3

4

5

Improvement [%]Figure 5. Correlation between estimated chord precisionand the improvement of F-measure.

sion of chord estimation for each piece (correlation coefficient r = 0.33). This indicates that refining the language
model also improves the pitch estimation.
Moreover, as shown in Fig. 6, major and minor chords
like those in Fig. 4 were obtained as emission probabilities
directly from music audio signals without any prior knowledge. This implies that frequently used chord types canbe inferred from music audio signals automatically, which
would be useful in music classification or similarity analysis. The performance in the supervised setting (65.5%)
was better than the performance obtained in the unsupervised settings. Since there exist published piano scoreswith chord labels, this setting is considered to be practical. Although this difference was statistically insignificant (standard error was about 1.5%), F-measures were improved for 25 pieces out of 30. Moreover, the improvement
exceeded 1% for 15 pieces. The example of pitch estimation shown in Fig. 7 indicates that insertion errors at low
pitches are reduced by integrating the language model. On
the other hand, insertion errors in total increased in the integrated model. This is because the constraint on harmonic
partials (shift-invariant) is too strong to appropriately estimate the spectrum of each pitch. As a result, the overtones
that should be expressed by a single pitch are expressedby multiple inappropriate pitches that do not exist in the
ground-truth.
There would be much room for improving the performance. The acoustic model has the strong constraint on
harmonic partials as mentioned above. This constraint canbe relaxed by introducing source-filter NMF [4], whichfurther decomposes the bases into sources corresponding
to pitches and filters corresponding to timbres. Our model
corresponds the case the number of filters is one, and increment of the number of filters would contribute to express difference in timbres (e.g., difference between thetimbre of high pitches and that of low pitches). The language model, on the other hand, can be refined by introducing other music theory such as keys. Some methods
that treat the relationship between keys and chords [27],Figure 7.
Estimated piano-rolls for MUSbk xmas5 ENSTDkCl. Integrating the language model
redeuced Insertion errors at low pitches.
or keys and notes [12], have been studied. Moreover, thelanguage model focus on reducing unmusical errors suchas insertion errors in adjacent pitches, and is difficult tocope with errors in octaves or overtones. Modeling transitions between notes (horizontal relations) will contributeto solve this problem and to improve the accuracy.
5. CONCLUSION
We presented a new statistical multipitch analyzer that cansimultaneously estimate pitches and chords from music audio signals. The proposed model consists of an acoustic
model (a variant of Bayesian NMF) and a language model(Bayesian HMM), and each model can make use of eachother’s information. The experimental results showed thepotential of the proposed method for unified music transcription and grammar induction from music audio signals.
On the other hand, each model has much room for performance improvement: the acoustic model has a strong constraint, and the language model is insufficient to express
music theory. Therefore, we plan to introduce a sourcefilter model as the acoustic model and to introduce the concept of key in the language model.
Our approach has a deep connection to language acquisition. In the field of natural language processing (NLP),
unsupervised grammar induction from a sequence of words
and unsupervised word segmentation for a sequence of characters have actively been studied [28,29]. Since our model
can directly infer musical grammars (e.g., concept of chords)
from either music scores (discrete symbols) or music audio
signals, the proposed technique is expected to be useful foran emerging topic of language acquisition from continuous
speech signals [30].
Acknowledgement: This study was partially supported by JST
OngaCREST Project, JSPS KAKENHI 24220006, 26700020,
26280089, 16H01744, and 15K16054, and Kayamori Foundation.‘’Proceedings of the 17th ISMIR Conference, New York City, USA, August 7-11, 2016
6. REFERENCES[1] P. Smaragdis and J. C. Brown. Non-negative matrix
factorization for polyphonic music transcription. In
IEEE WASPAA, pages 177–180, 2003.
[2] K. Ohanlon, H. Nagano, N. Keriven, and M. Plumbley. An iterative thresholding approach to L0 sparse
hellinger NMF. In ICASSP, pages 4737–4741, 2016.
[3] M. Hoffman, D. M. Blei, and P. R. Cook. Bayesiannonparametric matrix factorization for recorded music.
In ICML, pages 439–446, 2010.
[4] T. Virtanen and A. Klapuri. Analysis of polyphonic audio using source-filter model and non-negative matrix
factorization. In Advances in models for acoustic processing, neural information processing systems workshop. Citeseer, 2006.
[5] J. L. Durrieu, G. Richard, B. David, and C. Févotte.
Source/filter model for unsupervised main melody extraction from polyphonic audio signals. IEEE TASLP,
18(3):564–575, 2010.
[6] D. Liang and M. Hoffman. Beta process non-negative
matrix factorization with stochastic structured meanfield variational inference. arXiv, 1411.1804, 2014.
[7] E. Vincent, N. Bertin, and R. Badeau. Adaptive harmonic spectral decomposition for multiple pitch estimation. IEEE TASLP, 18(3):528–537, 2010.
[8] G. E. Poliner and D. P. Ellis. A discriminative model
for polyphonic piano transcription. EURASIP Journal
on Applied Signal Processing, 2007.
[9] A. T. Cemgil. Bayesian inference for nonnegative matrix factorisation models. Computational Intelligence
and Neuroscience, 2009.
[10] M. Hamanaka, K. Hirata, and S. Tojo. Implementing agenerative theory of tonal music. Journal of New Music
Research, 35(4):249–277, 2006.
[11] E. Nakamura, M. Hamanaka, K. Hirata, and K. Yoshii.
Tree-structured probabilistic model of monophonicwritten music based on the generative theory of tonal
music. In ICASSP, 2016.
[12] D. Hu and L. K. Saul. A probabilistic topic model for
unsupervised learning of musical key-profiles. In ISMIR, pages 441–446, 2009.
[13] R. Jackendoff and F. Lerdahl. A generative theory of
tonal music. MIT Press, 1985.
[14] M. Rocher, T.and Robine, P. Hanna, and R. Strandh.
Dynamic Chord Analysis for Symbolic Music. Ann Arbor, MI: MPublishing, University of Michigan Library,
2009.
[15] A. Sheh and D. P. Ellis. Chord segmentation and recognition using EM-trained hidden Markov models. In ISMIR, pages 185–191, 2003.
[16] S. Maruo, K. Yoshii, K. Itoyama, M. Mauch, and
M. Goto. A feedback framework for improved chord
recognition based on NMF-based approximate notetranscription. In ICASSP, pages 196–200, 2015.

315[17] Y. Ueda, Y. Uchiyama, T. Nishimoto, N. Ono, and
S. Sagayama. HMM-based approach for automaticchord detection using refined acoustic features. In
ICASSP, pages 5518–5521, 2010.
[18] S. Raczynski, E. Vincent, F. Bimbot, and S. Sagayama.
Multiple pitch transcription using DBN-based musicological models. In ISMIR, pages 363–368, 2010.
[19] S. A. Raczynski, E. Vincent, and S. Sagayama. Dynamic bayesian networks for symbolic polyphonicpitch modeling. IEEE TASLP, 21(9):1830–1840, 2013.
[20] S. Sigtia, E. Benetos, and S. Dixon. An end-to-end neural network for polyphonic piano music transcription.
IEEE TASLP, 24(5):927–939, 2016.
[21] S. Sigtia, E. Benetos, S. Cherla, T. Weyde, A. Garcez,
and S. Dixon. An RNN-based music language model
for improving automatic music transcription. In ISMIR,
pages 53–58, 2014.
[22] A. T. Cemgil and O. Dikmen. Conjugate Gamma
Markov random fields for modelling nonstationary
sources. In Independent Component Analysis and Signal Separation, pages 697–705. Springer, 2007.
[23] M. Davy and S. J. Godsill. Bayesian harmonic models for musical signal analysis. Bayesian Statistics,(7):105–124, 2003.
[24] V. Emiya, R. Badeau, and B. David. Multipitch estimation of piano sounds using a new probabilistic spectral
smoothness principle. IEEE TASLP, 18(6):1643–1654,
2010.
[25] C. Schörkhuber, A. Klapuri, N. Holighaus, and
M. Dörfler. A Matlab toolbox for efficient perfectreconstruction time-frequency transforms with logfrequency resolution. In Audio Engineering Society
Conference, 2014.
[26] D. Fitzgerald. Harmonic/percussive separation using
median filtering. In DAFx, pages 1–4, 2010.
[27] K. Lee and M. Slaney. Acoustic chord transcription and
key extraction from audio using key-dependent HMMs
trained on synthesized audio. IEEE TASLP, 16(2):291–
301, 2008.
[28] M. Johnson. Using adaptor grammars to identify synergies in the unsupervised acquisition of linguistic structure. In Proceedings of the 46th Annual Meeting of the
Association of Computational Linguistics, pages 398–
406, 2008.
[29] D. Mochihashi, T. Yamada, and N. Ueda. Bayesian unsupervised word segmentation with nested Pitman-Yor
language modeling. In ACL, pages 100–108. Association for Computational Linguistics, 2009.
[30] T. Taniguchi and S. Nagasaka. Double articulation analyzer for unsegmented human motion using PitmanYor language model and infinite hidden markov model.
In IEEE/SICE International Symposium on System Integration, pages 250–255. IEEE, 2011.