Proceedings SMC 2016 | 31.8. - 3.9.2016, Hamburg, Germany

RHYTHM TRANSCRIPTION OF POLYPHONIC MIDI PERFORMANCESBASED ON A MERGED-OUTPUT HMM FOR MULTIPLE VOICESEita Nakamura
Kyoto University
enakamura@sap.ist.i.kyoto-u.ac.jp

Kazuyoshi Yoshii
Kyoto University
yoshii@kuis.kyoto-u.ac.jp

ABSTRACT

Shigeki Sagayama
Meiji University
sagayama@meiji.ac.jp

with a latent variable corresponding to time-varying tempos [10, 14, 15].
When HMMs are used for modelling polyphonic music,
we immediately face the problem of score representation.
A polyphonic score has multilayer structure, where concurrently sounding notes are grouped into several streams
or, in music terminology, voices 1 . A conventional way
is to represent a polyphonic score as a linear sequenceof chords [7]. However, this representation may not retain sequential regularities within voices, such as thosein polyrhythmic scores. Furthermore, properties of music performance, like the phenomenon of loose synchrony
between voices [17, 18], cannot be captured without explicitly modelling the multiple-voice structure.
The purpose of this paper is to construct a statistical model for rhythm transcription that can describe themultiple-voice structure of polyphonic music scores andperformances. We construct a model that describes polyphonic performances as merged outputs from multiple
component HMMs, each of which describes the generative
process of music scores and performances of one voice.
Our model is based on the merged-output HMM [19, 20],
which has been developed to describe, in an event-driven
manner, symbolic data of polyphonic music. We derive an
efficient inference algorithm that can simultaneously separate performed notes into voices and estimate their notevalues. The proposed model is compared with previously
studied HMM-based models by evaluating the accuracy ofrhythm transcription for piano performances. A complete
model description and extended evaluation results will bepresented in our forthcoming paper [23].
The main contribution of this study is the construction ofa rhythm-transcription algorithm that can explicitly handle multiple voices with guaranteed optimality. A statistical model with multiple-voice structure based on twodimensional probabilistic context-free grammar (PCFG)
models has been studied [11,12], but the algorithms developed in those studies had to use provided voice information
or a pruning technique that would sacrifice optimality.

This paper presents a statistical method of rhythm transcription that estimates the quantised durations (note values) of the musical notes in a polyphonic MIDI performance (e.g. piano) signal. Hidden Markov models(HMMs) have been used in rhythm transcription to combine a model for music scores and a model describing the
temporal fluctuations in music performances. However,when applied to polyphonic music, conventional HMMs
have a problem that they are based on representation ofpolyphonic scores as linear sequences of chords and thus
cannot properly describe the structure of multiple voices.
We propose a statistical model in which each voice is described with an HMM and polyphonic performances aredescribed as merged outputs from multiple HMMs, basedon the framework of merged-output HMM. We develop arhythm-transcription algorithm based on this model using
an efficient Viterbi algorithm. Evaluation results showedthat the proposed model outperformed previously studied
HMMs for rhythm transcription of polyrhythmic performances.
1. INTRODUCTION
Music transcription is a fundamental problem in music information processing, requiring the extraction of pitch andrhythm information from music audio signals. There havebeen many studies on converting a music audio signal intoa piano-roll representation based on acoustic modelling ofmusical sound [1, 2]. To obtain a music score, we must
recognise quantised note lengths (or note values) of themusical notes in piano rolls. For this purpose, many studies have been devoted to solving the problem of converting MIDI performances to music scores, which is calledrhythm transcription or quantisation [3–12]. In accordancewith the general trend, statistical modelling has been gathering attention recently in this field.
Hidden Markov models (HMMs) [13] are the most popular models used in recent studies on rhythm transcription [5–10]. Indeed a monophonic score, when represented
as a series of musical notes, can naturally be described witha Markov model. In addition, temporal fluctuations in performances can be described by a continuous-space HMM

2. RELATED WORKIn this section, we review previous HMM-based models
for rhythm transcription and discuss the problem of polyphonic extensions.

Copyright: c 2016 Eita Nakamura et al. This is an open-access article distributed
under the terms of the Creative Commons Attribution 3.0 Unported License, which
permits unrestricted use, distribution, and reproduction in any medium, provided

1 In this paper, a ‘voice’ means a unit stream of musical notes that cancontain chords.

the original author and source are credited.

338

Proceedings SMC 2016 | 31.8. - 3.9.2016, Hamburg, Germany

b 2
&bb4

œ œ

? b b 24 œbNote model

œ œ
œœ

˙
œ

j j j jœ œ œ œ œœœœ
Figure 2. A polyrhythmic passage (Chopin’s Fantaisie Impromptu) represented as a sequence of chords.

Metrical model

Figure 1. Two different representations of a music scorein previously proposed HMMs.

State representation of merged-output HMM

2.1 HMM-Based Models for Monophonic Music
HMMs for rhythm transcription usually consist of two
component models; a score model describing the probability of a score and a performance model describing theprobability of a performance given a score. HMMs in previous studies [5–10] can be classified into two groups according to the way the score model describes the sequenceof notes. In one class of HMMs for rhythm transcription,
which we call note HMMs, a score is represented as a sequence of note values and described with a Markov model(Fig. 1) [5,6]. To describe the temporal fluctuations in performances, one introduces a latent variable corresponding
to a (local) tempo that is also described with a Markov
model. An observed duration is described as a product of
the note value and the tempo that is exposed to noise of
onset times.
In another class of HMMs, which we call metrical
HMMs, a different description is used for the score model[8–10]. Instead of a Markov model of note values, aMarkov process on a grid space representing beat positionsof a unit interval, such as a bar, is considered (Fig. 1). The
note values are given as differences between successive
beat positions. The same performance model as in note
HMMs can be used. Incorporation of the metre structure isan advantage of metrical HMMs.

s

1/2

1

2

1

1

2

2

1

i(1)

i0

(1)

i1

(1)

i1

(1)

i2

(1)

i3

(1)

i3

(1)

i3

(1)

i4

i(2)

i0

(2)

i0

(2)

i1

(2)

i1

(2)

i1

(2)

i2

(2)

i3

(2)

i3

HMM 1 i(1)
0

i1

(1)

i3

(1)

HMM 2 i(2)0Outputs

i2

(1)

(2)

i1
x1

x2

x3

x4

(1)
(2)

(1)

i4
i2

(2)

i3

(2)

x5

x6

x7

Figure 3. A schematic illustration of the merged-output(1)(2)
HMM. The symbols i0 and i0 represent auxiliary states
to define the initial transitions.
to be provided for evaluations. Takamune et al. state that
this problem is solved using the generalised LR parser [12].
Although detailed explanations are lacking, their method
uses pruning and its optimality is not guaranteed.
Although the above two descriptions of polyphonic
scores are both logically possible, there are instances inwhich models based on the simplified representation cannot describe the nature of polyphonic music well. First,
complex polyphonic scores such as polyrhythmic scoresare forced to have unrealistically small probabilities. Thisis because such scores consist of rare rhythms in the simplified representation even if the component voices havecommon rhythms (Fig. 2). Second, the phenomenon ofloose synchrony between voices (e.g. two hands in piano
performances [17]), called voice asynchrony, cannot be
described. Indeed, the importance of incorporating themultiple-voice structure in describing polyphonic music is
well-established in studies on score-performance matching [17, 18]. The situation calls for a similar treatment ofmultiple voices for polyphonic rhythm transcription.

2.2 Polyphonic Extensions
There are two directions of polyphonic extensions: using asimplified representation of polyphonic scores or using an
extended model describing multiple voices. The first direction is based on a fact that any polyphonic score can be represented as a sequence of chords or, more precisely, ‘note
clusters’ consisting of one or more notes as far as only onsets are concerned. For note HMMs, chordal notes can be
represented as self-transitions in the score model (Fig. 1)
and their inter-onset intervals (IOIs) can be described witha probability distribution with a peak at zero [7]. Similar
extensions are possible for metrical HMMs.
For the second direction, a PCFG model has been extended to describe the multiple-voice structure of scores[11]. In addition to the divisions of a time interval, duplications of intervals into two voices are considered. Unfortunately, a tractable inference algorithm could not be obtained for this model, and the correct voice information had

2.3 Merged-Output HMM
Recently merged-output HMM has been proposed as an
HMM-based model for describing symbolic signals ofpolyphonic music with multiple voices. In the model, eachvoice is described with an HMM and the total signal isrepresented as merged outputs from these HMMs (Fig. 3).
The merged-output HMM can be seen as a variant of factorial HMM [21]. To appropriately describe the nature ofsymbolic signals and capture sequential regularities withineach voice, only one of the component HMMs is involved
with each output in a merged-output HMM, whereas allcomponent HMMs contribute to every output in a standard

339

Proceedings SMC 2016 | 31.8. - 3.9.2016, Hamburg, Germany

factorial HMM. Basic inference algorithms for mergedoutput HMMs have been provided in our previous studies [19, 20].

is described with a normal distribution as(s)

We present an HMM-based model for rhythm transcription that describes polyphonic performances with multiplevoice structure. Given a polyphonic MIDI performance
signal, the model can simultaneously separate performed
notes into voices and estimate their note values. To construct a model based on a previously studied HMM [7] andapply the framework of merged-output HMM [19, 20], we
address the following issues: (1) pitches should be explicitly modelled to appropriately describe voices; (2) tempos
of multiple voices should be bound to assure loose synchrony between voices. After explaining the note HMM indetail in Sec. 3.1, a model satisfying these requirements ispresented in Sec. 3.2, and a sketch of inference algorithm
is given in Sec. 3.3.
A music score is specified by multiple sequences, corresponding to voices, of pitches and note values. Since
polyrhythm and voice asynchrony typically involve twovoices, we formulate the model with two voices indexed
by a variable s = 1, 2. A MIDI performance signal isspecified by a sequence of pitches and onset times.

(s)

(s)

(s)

rn(s) |rn 1 ⇠ Cat(⇡ (s) )
rn

(s)

pn

(s)

2
v ),

(5)

(6)

(s)

p0 ,p

(n = 2, . . . , Ns +1),

(7)

1

(s)

1

pn

1 ,p

)p is the (stationary) transition probabil-ity. These parameters are to be learned from music data.
The above model can be summarised as an autoregressive
HMM, which we call a voice HMM, with hidden states(r (s) , v (s) ) and outputs (p(s) , t(s) ). Although so far the
probabilities of pitches are independent of other variables,
they will be significant once multiple voice HMMs are
merged and the posterior probabilities are inferred.
3.2 Model for Multiple Voices
We combine the multiple voice HMMs in Sec. 3.1 using the framework of merged-output HMMs [19]. Simply speaking, the sequence of merged outputs is obtainedby gathering the outputs of the voice HMMs and sortingthem according to onset times. To derive inference algorithms that are computationally tractable, however, weshould formulate a model that outputs notes incrementallyin the order of observations. This can be done by intro+1
ducing stochastic variables s = (sn )N
n=1 , which indicatethat the n-th observed note belongs to voice sn , with thefollowing probability:

1 ,ris the (stationary) transition probability. Chordal notes are
represented as self-transitions of note values (Fig. 1). The
probability values are to be learned from music data.
To describe the temporal fluctuations, we introduce a(s)
tempo variable, denoted by vn , that describes the local
tempo for the n-th note. To represent the variation of tempos, we put a Gaussian Markov process on the logarithmof the tempo variables as
(s)

(s)

p0

✓ (s) = (✓ (s)(s)

ln vn(s) |ln vn 1 ⇠ N(ln vn 1 ,2t ).

where ✓ (s) = (✓ (s) )p is the initial probability, and

(2)

rn

(s)

pn

where Cat denotes the categorical distribution, ⇡ini =
(s)
(s)
(s)
(⇡ini,r )r is the initial probability, and ⇡ (s) = (⇡ (s) )r
1

(s)

(s)

(s)

1

rn

(s)

p(s)
n |pn 1 ⇠ Cat(✓ (s) )(1)(n = 2, . . . , Ns ),

(s)

p0

For each voice we first construct a model based on the one
presented in a previous study [7]. Let Ns be the number(s)of score notes in voice s and let rn denote the note value(s)sof the n-th note. The note values r (s) = (rn )N
n=1 aregenerated by a Markov chain with the probability given as
(s)

(s)

p1 |p0 ⇠ Cat(✓ (s) ),3.1 Model for Each Voice

(s)

(s)

The measured values of the parameters are t = 0.02 s
and = 0.0101 s [15] (the value of v will be explainedlater). Remarks should be made here: First, the number ofobserved onsets must be Ns +1 so that there are Ns IOIs
corresponding to Ns score notes. Second, we do not put(s)a distribution on the onset time of the first note t1 because we formulate the model to be invariant under timetranslations and this value would not affect any results of(s)sinference. We will use the notation v (s) = (vn )N
n=1 and(s) Ns +1(s)t = (tn )n=1 .
Finally we describe the generation of pitches p(s) =(s)s +1(pn )Nn=0 as a Markov chain (we introduce an auxiliary(s)symbol p0 for later convenience). The probabilities are

3. PROPOSED MODEL

r1 ⇠ Cat(⇡ini ),

(s)

t(s)
n |tn 1 , vn 1 , rn 1 ⇠ N(tn 1 + rn 1 vn 1 ;sn ⇠ Ber(↵1 , ↵2 ),(8)

where Ber is the Bernoulli distribution. ↵sn represents
how likely the n-th note is generated from the HMM ofvoice sn and, to improve the results of voice separation, we
put on the parameter conditional dependence on the lowest
and highest pitches of simultaneously sounding notes.
If voice sn is chosen, then the HMM of voice sn outputs a note, and the hidden state of the other voice
HMM is unchanged. Such a model can be describedwith an HMM with a state space labelled by kn =(1) (1) (1) (2) (2) (2)(sn , pn , rn , tn , pn , rn , tn , vn ). Here we have a single tempo variable vn that is shared by the two voices in order to assure loose synchrony between them. P (kn |kn 1 ),(3)

where N denotes the normal distribution. If the (n 1)-thand n-th notes belong to a chord, their IOI approximately
obeys an exponential distribution [15] and the probability(s)of the onset time of the n-th note, denoted by tn , is thengiven as
(s)
t(s)
(4)
n |tn 1 ⇠ Exp( ),where Exp denotes the exponential distribution and is(s)(s)
the scale parameter. Otherwise, tn
tn 1 has a dura(s)tion corresponding to note value rn 1 and the probability

340

Proceedings SMC 2016 | 31.8. - 3.9.2016, Hamburg, Germany

for n2, is given as(sn )(sn ) (sn ) (sn )n)↵sn P (vn |vn 1 )A (sn ) (sn ) (p(sn , tn |pn 1 , tn 1 ; vn 1 )
r
r

·

h

n

1 n

sn 1 r (2) r (2) p(2) p(2)
n 1 n
n 1 n

(2)

(tn 1

where we have defined
(s)

A (s)
rn

(s)

(s)
1 rn

rn

(s)

(s)
(p(s)
n , tn |pn 1 , tn 1 ; vn 1 )(s)= ⇡ (s)it(2)n ) + (1 $ 2) , (9)

(s)

(s)
1 ,rn

✓ (s)
pn

(s)

(s)
1 ,pn

(s)

P (t(s)n |tn 1 , vn 1 , rn 1 )

(10)

and
denotes Kronecker’s delta for discrete variables
and Dirac’s delta function for continuous variables.
The probability P (vn |vn 1 ) is defined in Eq. (3), and(s ) (s )(s )P (tn n |tn n1 , vn , rn n ) is defined in Eqs. (4) and (5).
(s)
For note values the initial probability is given as r1 ⇠(s)Cat(⇡ini ), and for pitches the initial probability is set as(1)(2)
in Eq. (6). The first onset times t1 and t1 do not have
distributions, as explained in Sec. 3.1, and we practically(1)(2)
set t1 = t1 = t1 where t1 is the first observed onset
time. Finally the output of the model is given asn)pn = p(sn ,n)tn = t(sn ,4. EVALUATION
4.1 Setup
We evaluated the proposed model by comparing the accuracy of its rhythm transcription with that of previously
studied models based on HMMs. Two data sets of MIDI
recordings of classical piano pieces were used. One(‘polyrhythmic’ data set) consisted of 18 performancesof 15 (excerpts of) pieces that contained 2 against 3 or
3 against 4 polyrhythmic passages, and the other (‘standard polyphony’ data set) consisted of 30 performancesof 22 pieces that did not contain polyrhythmic passages.
Pieces by various composers, ranging from J. S. Bach to
Debussy, were chosen and the players were also various:
Some of the performances were taken from the PEDB
database [22], a few were performances we recorded, andthe rest was taken from public domain websites.
All normal, dotted, and triplet note values ranging fromthe whole note to the 32nd note were used as candidate
note values. The transition and initial probabilities of thenote values and pitches, and the value of ↵s , were learned
from a data set of classical piano scores that had no overlap
with the test data. For the tempo variable, we discretised vn
into 50 values logarithmically equally spaced in the rangeof 0.3 to 1.5 sec per quarter note (corresponding to 200
BPM and 40 BPM). The standard deviation in Eq. (3) was
set as v = 1.08, using the value in [15] as a reference.
For comparison, we implemented the note HMM [6]and the metrical HMM [8] that is extended to handle
polyphony. The parameters of the score models were also
trained with the same score dataset. The performancemodel was the same as that for the proposed model.
We used as an evaluation measure the rhythm correction
ratio, i.e., the ratio of the smallest number of edit operations needed to correct the estimated result to the numberof notes in the data. In addition to note-wise correction(shift operation), the scaling operation applied for a subsequence of note values was included. This is because there
is arbitrariness in choosing the unit of note values: For example, a quarter note played in a tempo of 60 BPM has thesame duration as a half note played in a tempo of 120 BPM.
The smallest number of necessary edit operations Ne canbe calculated by a dynamic programming similar to thatused in computation of the Levenshtein distance (see ourforthcoming paper [23] for details). The rhythm correction ratio R is then given as R = Ne /N . When separated
voices are given, we can apply the above editing of notevalues for each voice and then the total rhythm correction
cost is the sum of the rhythm correction costs in all voices.

(11)

and thus the complete-data probability is written asYn)
P (k, p, t) =P (kn |kn 1 ) pn p(sn ) (tn t(sn ). (12)nrepresented by a variable describing the historical information of voices associated to notes, as suggested in [20].
Extending the formalism of introducing a latent variable todescribe this information, we can derive an efficient algorithm. Details will be given in our forthcoming paper [23].
We have confirmed that this algorithm can be executed ina standard modern computer environment with a practicaltime (within a few hours for a performance with hundredsof notes).

n

N = N1 + N2 denotes the total number of scorenotes, and the following notations will be used: v =
N +1
N +1
(vn )Nn=1 , p = (pn )n=1 , t = (tn )n=1 , and k =N +1(kn )n=1 . Note that whereas p and t are observed quantities, p(1) , p(2) , t(1) , t(2) are not because we cannot directly
observe the voice information encrypted in s.
3.3 Inference Algorithm
Rhythm transcription based on the proposed model can be
performed by estimating the most probable hidden state sequence k̂ given the observations (p, t). Once k̂ is obtained,
we can extract the voice information ŝ and the note valuesr̂ (1) and r̂ (2) . These are the result of voice separation andrhythm transcription.
The maximisation of the probability P (k|p, t) can be inprinciple done with the Viterbi algorithm [13]. However,
due to the complexity of our model, we need refinements
to the standard Viterbi algorithm to derive a computationally tractable algorithm. First, since the state space of themerged-output HMM in Sec. 3.2 involve both discrete andcontinuous variables, an exact inference is not computationally tractable. To solve this problem, we discretise thetempo variable in a range that is common in music practice. Other continuous variables t, t(1) , and t(2) can takeonly values of observed onset times and thus can, in effect,
be treated as discrete variables.
Second, it appears that a Viterbi algorithm derived in theway proposed in [19] has rather large computational costfor the present model and in practice difficult to execute.
The large computational cost derives from the fact that
we need to model pitches and onset times for the voice
HMMs. This problem can be reduced by noting that thepitch and onset time are observed quantities and can be341

Proceedings SMC 2016 | 31.8. - 3.9.2016, Hamburg, Germany

Data set
Polyrhythmic

Model
R [%]
Proposed
16.0 ± 3.6
Note HMM [6]28.9 ± 4.9
Metrical HMM [8] 34.1 ± 5.0
Standard polyphony Proposed
7.9 ± 1.3
Note HMM [6]7.0 ± 1.3
Metrical HMM [8] 7.9 ± 1.4

An important future direction of developing advanced
transcription techniques is to capture the phrase or motivic
structure of music. Recognition of offsets and articulations
and detection of ornaments are challenging problems. The
treatment of voice structure is a fundamental problem forthese issues, and the results of this study may be applicable
to solving these problems.

Table 1. Average rhythm correction rates R with standard
errors. Lower is better.

Acknowledgments
This work is partially supported by JSPS KAKENHI Nos.
24220006, 26240025, 26280089, 26700020, 15K16054,
16H01744 and 16J05486, JST OngaCREST Project andKayamori Foundation. E. Nakamura is supported by the
JSPS fellowship program.

4.2 Results
Results in Table 1 show that the proposed model clearly
outperformed the other models for performances with
polyphonic passages.
Fig. 4 shows an example thata polyrhythmic passage is successfully transcribed withthe proposed model with minor errors 2 . We see thatthe proposed model correctly recognised the 3 against 4
polyrhythms. On the contrary, the Note HMM did notrecognise the polyrhythms (cf. Fig. 2) and had frequent errors in chord clustering.
For performances in standard polyphony, on the other
hand, the note HMM was slightly better than the proposedmodel and the metrical HMM. Presumably, the main reason is that the rhythmic pattern in the reduced sequenceof chords is often simpler than that of melody/chords ineach voice in the case of standard polyphony because of
the principle of complementary rhythm [24]. In particular, notes/chords in a voice can have tied note values thatare not contained in our candidate list (e.g. quarter note
+ 16th note value), which can also appear as a result ofincorrect voice separation (Fig. 5). It is also observed thatthe transcription by the merged-output HMM can produce
desynchronised cumulative note values in different voices.
This is due to the lack of constraints to assure the matchingof these cumulative note values and the simplification of
independent voice HMMs. Further improvements are expected by incorporating such constraints and interactions
between voices into the model.
For the note HMM and the proposed model, there weregrammatically wrong sequences of note values, for example, triplets that appear in single or two notes without completing a unit of beat. This can be avoided with a refinedscore model with beat/bar structure [6, 11]. On the other
hand, these grammatical errors were not observed in thetranscriptions by the metrical HMM owing to the explicitly included metrical structure.
5. CONCLUSION
To develop a rhythm transcription algorithm that captures
the voice structure, we constructed a stochastic model ofmusical score and performance using the framework ofmerged-output HMMs. The evaluation results confirmedthat the proposed algorithm worked better for polyrhythmic performances than the previously proposed HMMbased algorithms.
2 Soundfiles and more examples are accessible in our demonstration web page: http://anonymous4721029.github.io/
demo.html

6. REFERENCES[1] A. Klapuri and M. Davy (eds.), Signal Processing
Methods for Music Transcription, Springer, 2006.
[2] E. Benetos et al., “Automatic Music Transcription:
Challenges and Future Directions,” J. Intelligent Information Systems, vol. 41, no. 3, pp. 407–434, 2013.
[3] H. Longuet-Higgins, Mental Processes: Studies in
Cognitive Science, MIT Press, 1987.
[4] P. Desain and H. Honing, “The Quantization of Musical Time: A Connectionist Approach,” Comp. Mus. J.,
vol. 13, no. 3, pp. 56–66, 1989.
[5] T. Otsuki et al., “Musical Rhythm Recognition Using
Hidden Markov Model (in Japanese),” J. Information
Processing Society of Japan, vol. 43, no. 2, pp. 245–
255, 2002.
[6] H. Takeda et al., “Hidden Markov Model for Automatic Transcription of MIDI Signals,” Proc. MMSP,
pp. 428–431, 2002.
[7] H. Takeda et al., “Rhythm and Tempo Analysis Toward
Automatic Music Transcription,” Proc. ICASSP, vol. 4,
pp. 1317–1320, 2007.
[8] C. Raphael, “Automated Rhythm Transcription,” Proc.
ISMIR, pp. 99–107, 2001.
[9] M. Hamanaka et al., “A Learning-Based Quantization:
Unsupervised Estimation of the Model Parameters,”
Proc. ICMC, pp. 369–372, 2003.
[10] A. Cemgil and B. Kappen, “Monte Carlo Methods for
Tempo Tracking and Rhythm Quantization,” J. Artificial Intelligence Res., vol. 18 no. 1, pp. 45–81, 2003.
[11] M. Tsuchiya et al., “Probabilistic Model of TwoDimensional Rhythm Tree Structure Representation
for Automatic Transcription of Polyphonic MIDI Signals,” Proc. APSIPA, pp. 1–6, 2013.
[12] N. Takamune et al., “Automatic Transcription fromMIDI Signals of Music Performance Using 2Dimensional LR Parser (in Japanese),” Tech. Rep. SIGMUS, vol. 2014-MUS-104, no. 7, pp. 1–6, 2014.
[13] L. Rabiner, “A Tutorial on Hidden Markov Models andSelected Applications in Speech Recognition,” Proc.
IEEE, vol. 77, no. 2, pp. 257–286, 1989.

342

Proceedings SMC 2016 | 31.8. - 3.9.2016, Hamburg, Germany

Correcttranscription(scaled by 2)

Note HMM

1

2

œ
œ
##
#œ œ œ œ œ œ& # # C œ #œ œ œ œ œ #œœ? #### C&

Proposed
model

(sec)

0

Input
performance

####

&

3

œ

œ

? ####

œœ

? ####œ œœ3œ œ œ œ
œ œ
3

3

œ
œ
# œ œ œ œ œ # œ # œ œ œ œ œœœ œœ œœ œ œ œ
œ
œ œ
3

3

####

œ

œ

œ

3

œ

œ

œ

3

#œ œ œ
œ œ

œ

3

3

œ œ
œ œ

#œ

#œ œ œ

œ

œœ
œ
œ œ
œ
œ œ

œ

3

œ œ œ œ œ œ œœ œ œ œ œœ œ œ œ
œ œ
œ œ
œ œ
œ
œ
œ

œ œ œ
3

3

3

3

nœ nœ œ œ œœ œ œœ œ œ œ nœ œ œœ œœ œ œ œ œ
œ
œ œ
œ
œ
œ

3

3

3

œ œ œ
œœœ
œ œ
œ œ œ
œ
œ œ œ œ œ
œ œ
œ
œ
œ

œ

œ

3

œœ

œœ

Figure 4. Transcription results of a polyrhythmic passage. For the result with the proposed model (merged-output HMM),
the staffs indicate the estimated voices.
0

(sec)

1

2

3

4

5

6

Input
performance

Correcttranscription
Proposedmodel(scaled by 2)

Note HMM(scaled by 2)

##
& # c

? ### c ‰
##
& #
##
& #
##
& #

∑

œ
œ œ œ œ œ œ œ œ œ œ œÓ‰ œ œ œ œ œœœ œ œ œ œ œ œ œ œ œ œ œ3œœ œ œ œ œ œ œ œ œ œ œœœ œ œ œ œ œ œ œ œ œ œ˙œ œ œ œ œ œ œ œ œ œœ œ œœ œ œœ œœ œ œ œ œ œ œ œ œ œœ œ œœ œ œœ œœœœ œœœ œœœœœœœœœ œ œ œ œ œ #œ œ
œ
J
J
3

w
œ
œœ œ œ

œ

œ œœ

œ #œ

œœ œ œ œ œ œ œœ œ # œœœœœœ œœ
œœœœœœœ œœ
œœœ

Figure 5. Transcription results of a standard polyphonic passage. For the result with the proposed model (merged-output
HMM), the staffs indicate the estimated voices.
[14] C. Raphael, “Automatic Segmentation of Acoustic Musical Signals Using Hidden Markov Models,” IEEE
Trans. on PAMI, vol. 21, no. 4, pp. 360–370, 1999.
[15] E. Nakamura et al., “A Stochastic Temporal Modelof Polyphonic MIDI Performance with Ornaments,” J.
New Music Res., vol. 44, no. 4, pp. 287–304, 2015.
[16] A. Cont, “A Coupled Duration-Focused Architecture
for Realtime Music to Score Alignment,” IEEE Trans.
on PAMI, vol. 32, no. 6, pp. 974–987, 2010.
[17] H. Heijink et al., “Data Processing in Music Performance Research: Using Structural Information to
Improve Score-Performance Matching,” Behavior Research Methods, Instruments, & Computers, vol. 32,
no. 4, pp. 546–554, 2000.
[18] B. Gingras and S. McAdams, “Improved ScorePerformance Matching Using Both Structural andTemporal Information from MIDI Recordings,” J. New
Music Res., vol. 40, no. 1, pp. 43–57, 2011.

with Ornaments, Desynchronized Voices, Repeats andSkips,” Proc. Joint ICMC|SMC 2014, pp. 1185–1192,
2014.
[20] E. Nakamura et al., “Merged-Output HMM for Piano
Fingering of Both Hands,” Proc. ISMIR, pp. 531–536,
2014.
[21] Z. Ghahramani and M. Jordan, “Factorial Hidden
Markov Models,” Machine Learning, vol. 29, pp. 245–
273, 1997.
[22] M. Hashida et al., “A New Music Database Describing
Deviation Information of Performance Expressions,”
Proc. ISMIR, pp. 489–494, 2008.
[23] E. Nakamura et al., in preparation.
[24] F. Salzer and C. Schachter, Counterpoint in Composition: The Study of Voice Leading, Columbia University
Press, 1989.

[19] E. Nakamura et al., “Merged-Output Hidden Markov
Model for Score Following of MIDI Performance343