TOWARDS COMPLETE POLYPHONIC MUSIC TRANSCRIPTION:
INTEGRATING MULTI-PITCH DETECTION AND RHYTHM QUANTIZATION
Eita Nakamura1 , Emmanouil Benetos2 , Kazuyoshi Yoshii1 , Simon Dixon2
1

2

Graduate School of Informatics, Kyoto University, Kyoto 606-8501, Japan
Centre for Digital Music, Queen Mary University of London, London E1 4NS, UK
ABSTRACT

Most work on automatic transcription produces “piano roll” datawith no musical interpretation of the rhythm or pitches. We present
a polyphonic transcription method that converts a music audio signal
into a human-readable musical score, by integrating multi-pitch detection and rhythm quantization methods. This integration is madedifficult by the fact that the multi-pitch detection produces erroneousnotes such as extra notes and introduces timing errors that are added
to temporal deviations due to musical expression. Thus, we proposea rhythm quantization method that can remove extra notes by extending the metrical hidden Markov model and optimize the model parameters. We also improve the note-tracking process of multi-pitch
detection by refining the treatment of repeated notes and adjustmentof onset times. Finally, we propose evaluation measures for transcribed scores. Systematic evaluations on commonly used classicalpiano data show that these treatments improve the performance oftranscription, which can be used as benchmarks for further studies.
Index Terms— Automatic transcription; multi-pitch detection;rhythm quantization; music signal analysis; statistical modelling.
1. INTRODUCTION
Automatic music transcription, or conversion of music audio signals
into musical scores, is a fundamental and challenging problem inmusic information processing [1, 2]. As musical notes in scores aredescribed with a pitch quantized in semitones and onset and offset
times quantized in musical units (score times), it is necessary to recognize this information from audio signals. In analogy with statistical speech recognition [3], one approach is to integrate a score modeland an acoustic model [4]. However, due to the huge number of possible combinations of pitches in chords, this approach is currently
infeasible for polyphonic music. A more popular approach is to separately carry out multi-pitch detection (quantization of pitch) andrhythm quantization (recognition of onset and offset score times).
Multi-pitch detection methods receive a polyphonic music audio
signal and output a list of notes (called note-track data) represented
by onset and offset times (in sec), pitch, and velocity, describing theconfiguration of pitches for each time frame. State-of-the-art approaches typically fall into two groups: spectrogram factorization
or deep learning. Spectrogram factorization methods decompose
an input spectrogram typically into a basis matrix (corresponding
to spectral templates of individual pitches or harmonic components)
This work is supported by JSPS KAKENHI (Nos. 24220006, 26280089,
26700020, 15K16054, 16H01744, 16H02917, 16K00501, and 16J05486)
and JST ACCEL No. JPMJAC1602. EN is supported by the JSPS Postdoctoral Research Fellowship and the long-term overseas research fund by the
Telecommunications Advancement Foundation. EB is supported by a UK
Royal Academy of Engineering Research Fellowship (grant no. RF/128).

ERB 200
frequency
bin 150

Polyphonicmusic audio
spectrogram(Mozart: Piano Sonata K331)

100
50
1
16

17

18

19

20

21

Time [s]Multi-pitch detection+ improved note tracking
16 (sec)

Note-track data

&
?

#

#

#

17
#

#

#

#

# 18

#

19

#

#

#

#

#

#

#

#
#

#

20

#

#
#

#

21
#

#

# #

#

Rhythm quantization+ removing extra notes##œ œ œ œœ œœ#2
Quantized MIDI data & # 4or? #### 2 œmusical score4œœœ œ œ œ œœ œœ œœ œœ œ
œ

œ

œ

œ

œ

˙

œ œ œ œ œœ œ œ œœœœœ œ œ œ œœ œ œ œœœœ œœ œ œ œœœ œ œ œŒ

Fig. 1. Integration of multi-pitch detection and rhythm quantization
for polyphonic transcription, with refinements on both parts.
and a component activation matrix (indicating active pitches overtime). These include non-negative matrix factorization (NMF), probabilistic latent component analysis (PLCA), and sparse coding [5–7].
Deep learning approaches for multi-pitch detection have used feedforward, recurrent, and convolutional neural networks [8, 9].
Rhythm quantization methods receive note-track data or performed MIDI data (human performance recorded by a MIDI device)and output quantized MIDI data in which notes are associated withquantized onset and offset score times (in beats). Onset score times
are usually estimated by removing temporal deviations in the input
data, and approaches based on hand-crafted rules [10, 11], statistical models [12–18], and a connectionist approach [19] have beenstudied. A recent study [18] has shown that methods based on hidden Markov models (HMMs) are currently state of the art. Especially, the metrical HMM [13, 14] has the advantage of being able toestimate the metre and bar lines and avoid grammatically incorrectscore representations (e.g. incomplete triplet notes). For recognitionof offset score times or note values, a method using Markov random
fields (MRFs) has achieved the current highest accuracy [20].
Given the recent progress of multi-pitch detection and rhythm
quantization methods, we study their integration for a complete polyphonic transcription (Fig. 1). For this, we refine the frame-based
multi-pitch detection part to provide a more musically meaningful
output that is useful for subsequent rhythm quantization. Since notetrack data typically contain erroneous notes, e.g. extra notes (false
positives) that are not included in the ground-truth score, a rhythm
quantization method that can reduce these errors is needed to avoid
accumulating errors, as emphasized in [21]. Another issue is to adapt
the parameters of rhythm quantization methods for note-track datathat contain timing errors caused by the impreciseness of multi-pitch
detection in addition to temporal deviations resulting from musicalexpression. Lastly, an evaluation methodology for the whole transcription process should be developed (see [22] for a recent attempt).

Polyphonic music audio

Multi-pitch detection

Rhythm quantization⎧⎨⎩

Multi-pitch analysis (Sec. 3.1)
Note tracking (Sec. 3.2)
Note-track data⎧
⎨ Onset rhythm quantization (Sec. 4.2)⎩

Note value recognition [20]Hand separation [26]Quantized MIDI data

Score typesetting

MuseScore 2 [24]Musical score (e.g. MusicXML, PDF)⎧
⎨ PitchOnset & offset time (in sec)

⎩ Velocity (strength)

⎧
Pitch
⎪
⎪
⎨ Onset & offset score time (in beat)
Velocity (strength)⎪⎪⎩ Time signatureHand-part/staff information

Fig. 2. Architecture of the proposed system.

bandwidth (ERB) spectrogram denoted as Vω,t , where ω stands forthe frequency index and t stands for the time index. The spectrogram has Ω = 250 filters, with frequencies linearly spaced between
5 Hz and 10.8 kHz on the ERB scale and has a 23 ms hop size.
In this work, the ERB spectrogram is used instead of a variable-Q
transform (VQT) spectrogram used in [7], since the former provides
a more compact representation with a better temporal resolution.
In the acoustic model, the input ERB spectrogram is approximated as a bivariate probability P (ω, t). This is in turn decomposed
into marginal probabilities for pitch, instrument source, and soundstate activations. The model is formulated as follows:X
P (ω, t) = P (t)P (ω|q, p, i)Pt (i|p)Pt (p)Pt (q|p), (1)q,p,iThe contributions of this study are as follows. First, we create acomplete system for polyphonic transcription, from audio to rhythmquantized musical score, which to our knowledge has not been attempted before in the literature. Second, we propose a novel method
for rhythm quantization to reduce extra notes in note-track data. To
incorporate top-down knowledge about musical notes like regularity
in time, a generative model (named noisy metrical HMM) is constructed as a mixture process of a metrical HMM [13, 14] describing
score-originated notes and a noise model describing the generationof extra notes. Third, we optimize the parameters for the rhythm
quantization methods and examine the effect. Fourth, we refine a supervised multi-pitch detection method based on PLCA [7] by introducing processes for onset-time adjustment and repeated-note detection. Finally, we propose measures for evaluating estimated scores
given ground-truth scores and report systematic evaluations on commonly used classical piano data [23], which can serve as benchmarks
for further studies. We find that all of the above treatments contributeto improving accuracies (or reducing errors) and the best case significantly outperforms systems using commercial software (MuseScore
2 [24] or Finale 2014 [25]) for rhythm quantization.

where p is the pitch index (p ∈ {1 = A0, . . . , 88 = C8});q ∈ {1, . . . , Q} is the sound-state index (with Q = 3, denoting
attack, sustain, and release); and i ∈ {1, . . . , I} is the instrumentsource index (with I =P8, here corresponding to 8 piano models).
P (t) corresponds toP (ω|q, p, i)ω Vω,t , a known quantity.
corresponds to a pre-learned 4-dimensional dictionary of spectral
templates per instrument i, pitch p, and sound state q. Pt (i|p)
refers to the instrument-source contribution for a specific pitch overtime, Pt (p) is the pitch activation, and Pt (q|p) is the sound-state
activation per pitch over time.
Unknown parameters Pt (i|p), Pt (p), and Pt (q|p) are iteratively
estimated using the expectation-maximization algorithm [28]. The
dictionary P (ω|q, p, i) is considered fixed and is not updated. Sparsity constraints are incorporated on Pt (p) and Pt (i|p), as in [7], tocontrol the polyphony level and the instrument-source contribution
in the resulting transcription. The output of the multi-pitch analysis is given by P (p, t) = P (t)Pt (p), which is the pitch activation
probability weighted by the magnitude of the spectrogram.
3.2. Note tracking

2. SYSTEM ARCHITECTURE
The architecture of the proposed polyphonic music transcription system is illustrated in Fig. 2. Although the architecture is applicable togeneral polyphonic music, some components are adapted for pianotranscription. The system has two main components: multi-pitch
detection and rhythm transcription (see also Sec. 1).
The multi-pitch detection part (Sec. 3) consists of multi-pitch
analysis (estimating multiple pitch activations for each time frame)
and note tracking (detecting notes identified by onset and offset
times, pitch, and velocity) and outputs note-track data. The rhythm
quantization part consists of onset rhythm quantization (inferring
the onset score times; Sec. 4) and note value recognition (inferring
the offset score times). For note value recognition, we use the MRF
method [20]. To include hand-part/staff information in quantized
MIDI data, we apply the hand separation method in [26].
Finally, to obtain human/machine-readable score notation (e.g.
MusicXML, PDF), we can apply the MIDI import function in score
typesetting software. Specifically, we use MuseScore 2 [24], which
has the ability to separate voices within each staff.
3. MULTI-PITCH DETECTION
3.1. Multi-pitch analysis
Our acoustic model is based on the work of [7], which performs
multi-pitch analysis through spectrogram factorization. The model
extends PLCA [27] and takes as input an equivalent rectangular

The note-tracking process converts the non-binary time-pitch representation of P (p, t) into a list of detected pitches, with an onset andoffset time. To do so, P (p, t) is thresholded and note events with aduration less than 30 ms are removed (following experiments on the
training set). Following this, we introduce a repeated-note detection
process. The process detects peaks in Vω,t for the time-frequency
regions corresponding to detected notes (we only use frequency binsthat correspond to the fundamental frequency of the detected note).
Any detected peaks in those regions indicate repeated notes, and thedetected note is subsequently split into smaller segments. A final
onset-time adjustment step slightly adjusts the start times of detected
notes by looking at detected onsets computed from Vω,t using thespectral flux feature. For each detected pitch, the process adjusts its
start time searching for detected onsets within a 50 ms window (thisprocess is applicable to musical instruments beyond the piano).
4. ONSET RHYTHM QUANTIZATION
4.1. Metrical HMM for onset rhythm quantization
We first review the metrical HMM [13,14], which consists of a scoremodel and a performance timing model. The score model generates
the beat position (onset score time relative to bar lines) of the n thnote bn ∈ {0, . . . , B−1} (B is the length of a bar) from the firstnote (n = 1) to the last one (n = N ). A binary variable (chordvariable) gn is used to describe whether the (n−1)th and n th notes
are in a chord (gn = CH) or not (gn = NC). The b1:N and g1:N aregenerated with the initial probability P (b1 , g1 ) and transition probability P (bn , gn |bn−1 ) with a constraint bn = bn−1 if gn = CH.
The difference between the (n−1)th and n th score times is given as
gn = CH;0,[bn−1 , bn , gn ] = bn − bn−1 ,gn = NC, bn > bn−1 ;b − bnn−1 + B, gn = NC, bn ≤ bn−1 .
The performance timing model generates onset times denotedby t1:N . To allow tempo variations, we introduce the local tempo
variables v1:N that are assumed to obey a Gaussian-Markov model:2
v1 = Gauss(vini , σiniv ),vn = Gauss(vn−1 , σv2 ),(2)

where Gauss(µ, Σ) denotes the Gaussian distribution with mean µ
and variance Σ, vini the initial (reference) tempo, σini v the standard
deviation describing the amount of global tempo variation, and σv
the standard deviation describing the amount of tempo changes. The
onset time of the n th note tn is determined stochastically by theprevious onset time tn−1 and variables vn−1 , bn−1 , bn , gn as [18]:(Gauss(tn−1 + vn−1 [bn−1 , bn , gn ], σt2 ), gn = NC;tn =(3)
Exp(tn−1 , λt ),gn = CH,
where Exp(x, λ) denotes the exponential distribution with scale parameter λ and support [x, ∞). For onset rhythm quantization, we
can infer b1:N , g1:N , and v1:N from given inputs t1:N , with theViterbi algorithm with discretization of the tempo variables.

Beat positionor score time
Local tempo(time-stretching rate)

q

b1
v1

e e q

b 2 b 3 b4v2 v 3v4qMetrical HMM(signal model)

b5

Onset-time
probability

sn

v5
Time

Noise model

Onset-time
probability

Time

MergedoutputTime

Score-originated notes

Extra notes

Fig. 3. Generation of onset times in the noisy metrical HMM.
Because duration and velocity are defined for positive numbers,
we here assume P (f |s) = IG(f ; as , bs ), where IG(x; a, b) =ba x−a−1 e−b/x /Γ(a) denotes the inverse-gamma distribution withshape parameter a and scale parameter b. (The formulation does notalter for the case of a more elaborate distribution.) The introductionof features can be seen as a modification to the probability αsn :Y
αsn → αs0 n = αsn
P (fn |sn )wf ,(8)
f : features

where the normal model has wf = 1. As the number of features weintroduce is arbitrary, it is reasonable to consider wf as a variable
that can be optimized by the maximum likelihood principle etc. In
this study, we optimize wf according to the error rate of transcription(see Sec. 5). An inference algorithm for the noisy metrical HMM canbe derived using a technique developed in [18].
5. EVALUATION

4.2. Noisy metrical HMM

5.1. Evaluation measures

The noisy metrical HMM is constructed by combining the metrical
HMM and a noise model. The noise model generates onset times asP∗ (tn |t0 ) = Gauss(tn ; t0 , σ∗2 ),(4)

where σ∗ is a standard deviation that is supposed to be larger than σt .
The reference time t0 will be set to t̃n introduced below. To constructa model combining the metrical HMM and the noise model, we introduce a binary variable sn ∈ {S, N} obeying a Bernoulli distribution: P (sn ) = αsn (αS + αN = 1). If sn = S, tn is generated according to the metrical HMM in Sec. 4.1; if sn = N, it is generated
according to Eq. (4). This process is described as a merged-output
HMM [18] with a state space indexed by zn = (sn , bn , gn , vn , t̃n )and the following transition and output probabilities (Fig. 3):
P (zn |zn−1 ) = δsn N αN δbn−1 bn δgn−1 gn δ(vn −vn−1 )δ(t̃n −t̃n−1 )+ δsn S αS P (bn , gn |bn−1 )P (vn |vn−1 )P (t̃n |t̃n−1 ),P (tn |zn ) = δsn S δ(tn − t̃n ) + δsn N P∗ (tn |t̃n ),

(5)
(6)

where δ denotes Kronecker’s delta for discrete arguments andDirac’s delta function for continuous arguments and P (t̃n |t̃n−1 )is given in Eq. (3). The t̃n memorizes the previous onset time fromthe signal model: t̃n = tn0 for the largest n0 < n with αsn0 = S.
The information of duration and velocity in note-track data canbe useful to identify extra notes since their distributions for extra
notes have smaller means and variances compared to the case forscore-originated notes. To utilize this information, we can extend
the model to describe the generation of features fn for each note.
(For notational simplicity, we use a unified notation fn to describe ageneral feature.) Their distribution is defined conditionally on sn as
P (fn = f ) = δsn S P (f |S) + δsn N P (f |N).

(7)

For evaluating the performance of the multi-pitch detection component of Sec. 3, we use the onset-based note-tracking metrics definedin [29], which are also used in the MIREX note-tracking public evaluations. These metrics assume that a note is correctly detected if its
pitch is same as the ground-truth pitch and its onset time is within±50 ms of the ground-truth onset time. Based on this rule, the precision Pn , recall Rn , and F-measure Fn metrics are defined.
Measures for evaluating transcribed musical scores in comparison to the ground-truth scores have been proposed in the context
of rhythm quantization [18, 20]. The rhythm correction cost (RCC)
is defined as the minimum number of scale and shift operations
for onset score times, which can be used for defining the onsettime error rate (ER) [18]. The offset-time ER can be defined bycounting incorrect offset score times relatively to the adjacent onset
score times [20]. To extend these ideas to the case with erroneousnotes, we first align the estimated score to the ground-truth score using a state-of-the-art music alignment method that can also identify
matched notes (i.e. correctly matched notes and notes with pitch errors), extra notes, and missing notes [30]. (A similar idea has been
discussed in [22].) We notate the number of notes in the groundtruth score by NGT , that in the estimated score by Nest , the numberof notes with pitch errors by Np , that of extra notes by Ne , and thatof missing notes by Nm , and define the number of matched notes as
Nmatch = NGT − Nm = Nest − Ne . Then we define the pitch
error rate Ep = Np /NGT , extra note rate Ee = Ne /Nest , missingnote rate Em = Nm /NGT , onset-time ER Eon = RCC/Nmatch ,and offset-time ER Eoff = No.e. /Nmatch , where the computationof RCC is explained in [18] and No.e. is the number of notes withan incorrect offset score time after normalization using the closest
onset score time (similarly as in [20]). We define the mean of thefive measures as the overall ER Eall .

Method
Pn Rn Fn p-value
HNMF [5]62.3 76.9 67.9 0.0034
PLCA-4D [7] 79.4 66.0 71.7 0.080
PLCA-4D-NT 77.9 68.9 72.8—
Table 1. Average accuracies (%) of multi-pitch detection on the
MAPS-ENSTDkCl dataset, comparing acoustic models. The lastcolumn shows the p-values of Fn with respect to PLCA-4D-NT.
MethodFinale 2014
MuseScore 2
MetHMM-def
MetHMM
NMetHMM

Ep
5.6
6.1
4.8
4.7
4.4

Em
24.2
26.1
25.2
25.4
28.6

Ee
18.3
16.9
15.7
16.3
13.3

Eon
53.3
39.7
29.6
23.6
21.6

Eoff
54.0
56.3
41.9
40.9
39.3

Eall

Eoff

Eall

p-value

Finale 201410.7 18.3 39.3 57.2 57.4
MuseScore 2 12.3 19.9 34.4 49.7 62.6
MetHMM-def 10.5 18.6 33.2 36.5 44.1
MetHMM
9.6 17.5 33.0 25.5 42.1
NMetHMM
7.2 20.8 19.8 24.1 41.2

Ep

36.6
35.8
28.6
25.5
22.6

< 10−5< 10−5< 10−50.00048—

Methodp-value−531.1 < 10
29.0 < 10−5
23.5 0.023
22.2
0.18
21.4
—

Table 2. Average error rates (%) of the whole transcription systems
on the MAPS-ENSTDkCl dataset, comparing rhythm quantization
methods applied on the outputs of the PLCA-4D-NT method. The
last column shows the p-values of Eall with respect to NMetHMM.

5.3. Results
Table 1 shows the accuracies of the multi-pitch detection methods.
We refer to the original PLCA-based method of [7] as PLCA-4D andthe note tracking additions of Sec. 3.2 as PLCA-4D-NT. The PLCA4D-NT method slightly outperforms the PLCA-4D method by about
1% in terms of the note-based F-measure, with a lower precisionand higher recall. The higher recall by the PLCA-4D-NT method
is considered more useful for the noisy metrical HMM, which can
reduce extra notes but cannot recover missing notes. The HNMF [5]method yields the highest recall but has the lowest F-measure.
Tables 2 and 3 show the results of evaluating the whole transcription method. For comparison, we run the metrical HMM with parameters taken from a previous study on rhythm quantization of performed MIDI data [18] (MetHMM-def) as well as the metrical HMM(MetHMM) and noisy metrical HMM (NMetHMM) with optimized
parameters. We also compared MusicXML outputs converted fromthe note-track data with two commercial software for score typesetting (MuseScore 2 [24] and Finale 2014 [25]). For both outputs fromthe PLCA-4D-NT and HNMF methods, the NMetHMM yields the

Ee

Eon

Table 3. Same as Table 2 but for outputs of the HNMF method [5].
ERB 200
frequencybin
150(Mozart: Piano Sonata K333)

Input spectrogram 10050
10Note-track databy PLCA-4D-NT

Transcribed scores
MuseScore 2

MetHMM-def

5.2. Experimental setup
For training the acoustic model in Sec. 3, we use a dictionary of spectral templates extracted from isolated note recordings in the MAPS
database [23]. The dictionary contains sound-state templates for 8
piano models found in the database, apart from the ‘ENSTDkCl’model, which is used for testing. The whole note range of the piano(A0 to C8) is used. Among the parameters of the symbolic modelin Sec. 4, P (b1 , g1 ), P (bn , gn |bn−1 ), vini , σini v , and σv are takenfrom a previous study [18] and αs , as , and bs are learned on the outputs of multi-pitch detection methods. The other parameters σ∗ , σt ,λt , and wf are optimized on the test data to maximize Eall .
For testing the transcription system, we use 30 piano recordingsin the ‘ENSTDkCl’ subset of the MAPS database [23], along with
their corresponding ground-truth note-track data and MusicXML
scores. For consistency with previous studies on multi-pitch detection, we only evaluate the first 30 s of each recording. For
comparison, we also run the multi-pitch detection method based on
harmonic NMF (HNMF) [5], which is based on adaptive NMF withpitch-specific spectra modelled as a weighted sum of narrowband
spectra, and apply our rhythm quantization method on its outputs.

Em

NMetHMM

Ground truth

2&?

0 (sec)

b

1

b

b

b
& b b 24 œ̇œŒ j Œ̇œŒ ‰ œ œ?b 2 ˙bb4

42bœ œ œ̇ œ ˙
‰J
œ
œ Œ
˙
˙

Œ

j
œ œœ

b 2
& b b 4 œœ ..

œ œœ œ
j‰ œb 3& b b 4 œœœœ œœ

?b 3 ˙
bb4

b

6
4

b

5

‰ jŒ
Œ
œ œ ‰ œJ n œœ Œbœœ Œ ˙ Œ

œœ ..

j
œ œœ

œ œœ œœ ..

œœ œ œœ
œ Œ Œ
˙.

Œ
œ

˙

œ
Œ

œœ œœ

œœ

Œ

œ

œ

7

œ

œ
˙

b

œœ ..

œ

œ
œœ
œ

œ nœ
b œœ

nœ
b œœ

9

b

œ
œ

œ

œœ

11

b

œ

b

b

b

Œ
œ.

‰
œ œ

œ
œ

Œ

12

b

b

b

b

b b

13

Extra note&‰ j
œ≈ . œ .œ œ œ≈ œ œ œ
J
Œ
∑

j
œœœ

œ œ œ œœ œ. œ œ œ. œ œ œ œ œ œ œ œ œ œ œ œ œ
˙
œ œ œ
Œ
∑
œ œ œ

œ

œ

12 Time [s]

10

b

Œ
Œ
?
?
œ œ .œ œœ œ &. œœœ œ
‰
‰
Œ
œ
Œ
∑

3

œœ

10
8

bb

œœ œjœ œ . œ n ˙
b ˙˙
œ œ Œ

œ

œœ

8
6

b

b

b 3
& b b 4 œœ ..
.
? b 3 ˙Ó .
bb4
Œ?b 2 ˙
bb4

3

b

b

œ

œ œ œ œ œœ œœœ œœœ œ œ œœœœ œ
œ
Œ
œ
œ
Œ
œ œ œ
J
œ
‰ ≈
3

œ œ œ œ
œ bœ
œ
Œ

œ œœ
œ ® œ œ œœœœœœ œœ œŒnœ œ œ œ œFig. 4. Example transcription results (Mozart: Piano Sonata K333
in the MAPS-ENSTDkCl dataset).
best average overall ER, which is significantly lower than the values
for commercial software. We find that the optimization of the parameters of the MetHMM consistently reduces ERs. Compared to
the MetHMM, the NMetHMM reduces all ERs except Em and its
effect is stronger for the higher-recall lower-precision outputs of the
HNMF method. In Fig. 4, we find that the NMetHMM correctly removes one extra note (G4 at 10.23 s) and corrects a misalignment ofchordal notes (E[4 and G4) found in the fourth bar of the transcribedscore by the MetHMM-def.
6. CONCLUSION
We have described integration of multi-pitch detection and rhythm
quantization methods for polyphonic music transcription. We have
improved the PLCA-based multi-pitch detection method by refining the note-tracking process and proposed a rhythm quantization
method based on the noisy metrical HMM aiming to remove extra
notes in note-track data, both of which led to better performance oftranscription. Optimizing the parameters of the metrical HMM describing temporal deviations was also effective to reduce errors.
Except for musically and acoustically simple cases, the transcribed scores obtained by our system contain musically incorrect
configurations of pitches and unplayable notes and are still far fromsatisfactory. The current noisy metrical HMM does not describe thepitch information. By incorporating a pitch model, those notes with
undesirable pitches are expected to be reduced. Correcting erroneousnotes in note-track data other than extra notes, i.e. pitch errors andmissing notes, is currently beyond the reach. Integration of a symbolic music language model with the acoustic model would be necessary for this. More thorough evaluations, including a subjective
one, are currently under investigation. There is also a need to examine the influence of alignment errors on the evaluation measures.

7. REFERENCES[1] A. Klapuri and M. Davy (eds.), Signal Processing Methods forMusic Transcription, Springer, 2006.
[2] E. Benetos, S. Dixon, D. Giannoulis, H. Kirchhoff, and A. Klapuri, “Automatic music transcription: Challenges and future
directions,” J. Intelligent Information Systems, vol. 41, no. 3,
pp. 407–434, 2013.
[3] S. Levinson, L. Rabiner, and M. Sondhi, “An introduction
to the application of the theory of probabilistic functions ofa Markov process to automatic speech recognition,” The Bell
Sys. Tech. J., vol. 62, no. 4, pp. 1035–1074, 1983.
[4] C. Raphael,“A graphical model for recognizing sung
melodies,” in Proc. ISMIR, 2005, pp. 658–663.
[5] E. Vincent, N. Bertin, and R. Badeau, “Adaptive harmonic
spectral decomposition for multiple pitch estimation,” IEEETASLP, vol. 18, no. 3, pp. 528–537, 2010.
[6] K. O’Hanlon and M. D. Plumbley, “Polyphonic piano transcription using non-negative matrix factorisation with group
sparsity,” in Proc. ICASSP, 2014, pp. 3112–3116.
[7] E. Benetos and T. Weyde, “An efficient temporally-constrained
probabilistic model for multiple-instrument music transcription,” in Proc. ISMIR, 2015, pp. 701–707.
[8] S. Sigtia, E. Benetos, and S. Dixon, “An end-to-end neuralnetwork for polyphonic piano music transcription,” IEEE/ACM
TASLP, vol. 24, no. 5, pp. 927–939, 2016.

[19] P. Desain and H. Honing, “The quantization of musical time:
A connectionist approach,” Comp. Mus. J., vol. 13, no. 3, pp.
56–66, 1989.
[20] E. Nakamura, K. Yoshii, and S. Dixon, “Note value recognition for piano transcription using Markov random fields,”
IEEE/ACM TASLP, vol. 25, no. 9, pp. 1542–1554, 2017.
[21] E. Kapanci and A. Pfeffer, “Signal-to-score music transcription
using graphical models,” in Proc. IJCAI, 2005, pp. 758–765.
[22] A. Cogliati and Z. Duan, “A metric for music notation transcription accuracy,” in Proc. ISMIR, 2017, pp. 407–413.
[23] V. Emiya, R. Badeau, and B. David, “Multipitch estimationof piano sounds using a new probabilistic spectral smoothness
principle,” IEEE TASLP, vol. 18, no. 6, pp. 1643–1654, 2010.
[24] MuseScore, “MuseScore 2,” https://musescore.org/en [online],
accessed on: Oct. 11, 2017.
[25] MakeMusic, “Finale 2014,” https://www.finalemusic.com [online], accessed on: Oct. 11, 2017.
[26] E. Nakamura, N. Ono, and S. Sagayama, “Merged-output
HMM for piano fingering of both hands,” in Proc. ISMIR,
2014, pp. 531–536.
[27] M. Shashanka, B. Raj, and P. Smaragdis, “Probabilistic latentvariable models as nonnegative factorizations,” Computational
Intelligence and Neuroscience, 2008, Article ID 947438.
[28] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. Royal
Stat. Soc., vol. 39, no. 1, pp. 1–38, 1977.

[9] R. Kelz, M. Dorfer, F. Korzeniowski, S. Böck, A. Arzt, and
G. Widmer, “On the potential of simple framewise approaches
to piano transcription,” in Proc. ISMIR, 2016, pp. 475–481.

[29] M. Bay, A. F. Ehmann, and J. S. Downie, “Evaluation ofmultiple-F0 estimation and tracking systems,” in Proc. ISMIR,
2009, pp. 315–320.

[10] H. Longuet-Higgins, Mental Processes: Studies in Cognitive
Science, MIT Press, 1987.

[30] E. Nakamura, K. Yoshii, and H. Katayose, “Performance error
detection and post-processing for fast and accurate symbolic
music alignment,” in Proc. ISMIR, 2017, pp. 347–353.

[11] D. Temperley and D. Sleator, “Modeling meter and harmony:
A preference-rule approach,” Comp. Mus. J., vol. 23, no. 1, pp.
10–27, 1999.
[12] A. T. Cemgil, P. Desain, and B. Kappen, “Rhythm quantization
for transcription,” Comp. Mus. J., vol. 24, no. 2, pp. 60–76,
2000.
[13] C. Raphael, “A hybrid graphical model for rhythmic parsing,”
Artificial Intelligence, vol. 137, pp. 217–238, 2002.
[14] M. Hamanaka, M. Goto, H. Asoh, and N. Otsu, “A learningbased quantization: Unsupervised estimation of the model parameters,” in Proc. ICMC, 2003, pp. 369–372.
[15] H. Takeda, T. Otsuki, N. Saito, M. Nakai, H. Shimodaira, and
S. Sagayama, “Hidden Markov model for automatic transcription of MIDI signals,” in Proc. MMSP, 2002, pp. 428–431.
[16] D. Temperley, “A unified probabilistic model for polyphonicmusic analysis,” J. New Music Res., vol. 38, no. 1, pp. 3–18,
2009.
[17] A. Cogliati, D. Temperley, and Z. Duan, “Transcribing human piano performances into music notation,” in Proc. ISMIR,
2016, pp. 758–764.
[18] E. Nakamura, K. Yoshii, and S. Sagayama, “Rhythm transcription of polyphonic piano music based on merged-output HMM
for multiple voices,” IEEE/ACM TASLP, vol. 25, no. 4, pp.
794–806, 2017.