AUTOREGRESSIVE HIDDEN SEMI-MARKOV MODEL OF SYMBOLIC
MUSIC PERFORMANCE FOR SCORE FOLLOWING
Eita Nakamura1
Philippe Cuvillier2
Arshia Cont2
1
Nobutaka Ono
Shigeki Sagayama3
1
National Institute of Informatics, Tokyo 101-8430, Japan
2
Inria MuTant Project-Team, Ircam/UPMC/CNRS UMR STMS, 75004 Paris, France
3
Meiji University, Tokyo 164-8525, Japaneita.nakamura@gmail.com, philippe.cuvillier@ircam.fr,
Arshia.Cont@ircam.fr
onono@nii.ac.jp, sagayama@meiji.ac.jp

ABSTRACTA stochastic model of symbolic (MIDI) performance ofpolyphonic scores is presented and applied to score following. Stochastic modelling has been one of the most successful strategies in this field. We describe the performance
as a hierarchical process of performer’s progression in thescore and the production of performed notes, and represent the process as an extension of the hidden semi-Markov
model. The model is compared with a previously studied
model based on hidden Markov model (HMM), and reasons are given that the present model is advantageous forscore following especially for scores with trills, tremolos,
and arpeggios. This is also confirmed empirically by comparing the accuracy of score following and analysing theerrors. We also provide a hybrid of this model and the
HMM-based model which is computationally more efficient and retains the advantages of the former model. The
present model yields one of the state-of-the-art score following algorithms for symbolic performance and can possibly be applicable for other music recognition problems.
1. INTRODUCTION
For the last thirty years the real-time matching of music
performance to the corresponding score (called score following) has been a popular field of study motivated byapplications such as automatic music accompaniment andscore-page turning system [1, 2, 3, 4, 5, 6, 7, 8]. We
study here score following of polyphonic symbolic (MIDI)
performance. A central problem in score following is toproperly capture the variety of music performance in acomputationally efficient manner. A commonly studied
way to capture this variety and develop an effective scorefollowing algorithm is to use stochastic models of music
performance (Sec. 2.1, see also [3]).
c Eita Nakamura, Philippe Cuvillier, Arshia Cont, Nobutaka Ono, Shigeki Sagayama. Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Attribution: Eita Nakamura, Philippe Cuvillier, Arshia Cont, Nobutaka Ono, Shigeki Sagayama.
“ Autoregressive Hidden Semi-Markov Model of Symbolic Music Performance for Score Following ”, 16th International Society for Music Information Retrieval Conference, 2015.

392

Hidden Markov models (HMMs) have been applied toscore following of symbolic performance and provided
currently best results [4, 7, 9]. In these models, a musical event in the score, i.e. note, chord, trill, etc., is represented as a state, and the performed notes are describedas outputs of an underlying state transition process. Memoryless statistical dependence is assumed for both output
and transition probabilities for the sake of computational
efficiency. Due to these simplifications the models cannot
well describe significant features of performance data suchas the number of performed notes per event and the total
duration of a trill.
Phenomenologically, music performance can be regarded as a hierarchical process of producing musicalnotes: The higher level describes performer’s progression
in the score in units of musical events, and the lower leveldescribes the production of individual notes [9, 10]. We
describe this process in terms of a hidden semi-Markov
model (HSMM) [11] with an autoregressive extension [12](Sec. 2) and incorporate the above features into the model.
With some simplifications, the model is reduced to a previously studied HMM [9]. We compare these models in
the informational and algorithmic aspects and argue thatthe present model is advantageous for score following especially for scores with trills, tremolos, and arpeggios(Sec. 3). Empirical confirmation of this fact is given bycomparing the accuracy of score following and analysing
the errors (Sec. 4). Finally remaining problems and future
prospects are discussed (Sec. 5).
2. AUTOREGRESSIVE HIDDEN SEMI-MARKOV
MODEL OF SYMBOLIC PERFORMANCE
2.1 Stochastic description of music performanceMusic performances based on a score have a wide variety because of indeterminacies inherent in musical score
descriptions and uncertainties in movements of performers and musical instruments. These indeterminacies anduncertainties are included in tempos, noise in onset times,
dynamics, articulations, ornaments, and also in the way ofmaking performance errors, repeats, and skips [7]. In orderto perform accurate and robust score following, we needProceedings of the 16th ISMIR Conference, Málaga, Spain, October 26-30, 2015
to incorporate (maybe implicit) rules into the algorithm tocapture this variety.
A way to do this is to construct a stochastic model ofmusic performance and describe those indeterminacies anduncertainties in terms of probability. A score-following algorithm can be developed as an inference problem of themodel. We shall take this approach in the following, which
has been proved to be successful in score following.

393

Score

Performance

2.2 Model of performer’s progression in the score
Let us present the model. We model music performance as
a combination of subprocesses in two levels. The higherlevel (top-level) process describes the performer’s progression in the score in units of musical events that are
well-ordered in performances without errors. We take achord (possibly arpeggiated), a trill/tremolo, a short appoggiatura, or an after note 1 as a unit and represent it witha state (top state). Let i label a top state. Then the performer’s progression can be described as successive transitions between these states denoted by i1:N = (i1 , . . . , iN )(N is the number of performed MIDI notes). We will use
the symbol n(= 1, . . . , N ) to index the performed notesthat are ordered according to the onset time, and in represents the corresponding musical event.
The probability P (i1:N ) describes statistical tendencies of performances. Simplifications are necessary toconstruct a performance model yielding a computationally tractable algorithm. A typical assumption is thatthe probability is decomposed into transition probabilities:
P (i1:N ) = ⇧N
n=1 P (in |in 1 ) (P (i1 |i0 ) ⌘ P (i1 ) denotes
the initial distribution). The probability P (j|i) represents
the relative frequency of straight progressions to the next
event (j = i + 1), insertions of events (j = i), deletions of an event (j = i + 2), and repeats or skips (if|j i 1| > 1). These probability values can be estimatedfrom performance data. With the assumption that P (i|j) is
only dependent on i j, the probability values have been
estimated with piano performance data in a previous study([7], Table 3).
2.3 Model of production of performed notes
The lower-level process describes the production of performed notes during each musical event. Because dynamics and articulations are generically highly indeterminate,
we focus on pitch and onset time which are denoted bypn and tn . For example, multiple notes are performed ata chord or a trill (Fig. 1). Note that whereas chords are
written in musical scores as simultaneous notes, performed
MIDI notes are serialised and never exactly simultaneous.
Thus pn is always a single pitch.
Let us first consider the number of performed notes per
event. For “chords” (meaning a set of all simultaneousnotes in the score), short appoggiaturas, and after notes,
the expected number of notes is determinate, but it can
1 Here ‘after notes’ are defined as grace notes that are played in precedence over the associated beat. A typical example is grace notes after atrill.

IOI3 IOI3 IOI2
IOI1(a) An arpeggiated chord.

Score
Performance
IOI3 IOI3

IOI1

IOI2

(b) Trill with preceding short-appoggiaturas and after notes.

Figure 1. Examples of musical events and performed
notes. The three types of time intervals IOI1, IOI2, andIOI3 are explained in the text.

be modified as a result of added or deleted notes by mistake. For trills and (unmeasured) tremolos, the numberof notes are indeterminate since the speed of ornaments
varies among realisations. We describe this situation witha probability distribution di (s) where s denotes the number of performed notes (⌃1
s=1 di (s) = 1). For example, thefunction di (s) peaks at the indicated number of notes when
event i is a chord. When event i is a one-note trill, the peakcan be written as speak' ⌫i v/ ttrill , where ttrill , ⌫i , andiv denote the average inter-onset time interval (IOI) of successive notes of a trill, the note value of event i, and the(inverse) tempo in units of “second per unit note value”.
Because currently we do not have a strong empirical basis
for determining the shape of di (s), we simply assume it isa normal distribution di (s) = N (s; speak, i ) with speakiigiven in Sec. 2.3, and leave i as an adjustable parameter.
Next the pitch of each performed note of event i can be
described with a probability Pipitch (p), which is assumed
to be independent for each note for the sake of computational efficiency. The probability values for incorrect
pitches represent the possibility and frequencies of pitch
errors. An approximate distribution of Pipitch (p) has been
estimated previously (Eq. (30) of [7]) with piano performance data, where the probability of pitch errors is assumed to be uniform for all score notes.
Finally we consider the description of onset times.
A natural assumption of time translational invariance requires the model to be only dependent of time intervals.
There are (at least) three different kinds of time intervals

394

Proceedings of the 16th ISMIR Conference, Málaga, Spain, October 26-30, 2015

relevant in locally describing onset times of music performance: (IOI1) The time interval between the first notesof succeeding events, which is typically the duration of anevent, (IOI2) the time interval between the first note of anevent and the last note of its previous event, and (IOI3) thetime interval between succeeding performed notes within
an event (Fig. 1). Assuming that the probability of thesetime intervals depends only on the current and previous
states for simplicity and computational efficiency, it has
the form P ( t|in 1 , in , v) ( = IOI1, IOI2, IOI3) wheret and v denote the relevant time interval and the tempo.
Based on the experience that time interval IOI3 is mostly
dependent on the relevant event and almost independentof tempo and other contexts, we further simplify the functional form as PIOI3 ( t|in ). Note that the time intervals
IOI1 and IOI2 are not independent quantities if we retain
all historical information on time, but they have different
importance when we take the Markovian description explained below.

pn 1

pn

pn+1

...

in 1
sn 1

in
sn

in+1
sn+1

...

...

tn 1

tn

tn+1

...

Figure 2. Graphical representation of the autoregressive hidden semi-Markov model of symbolic music performance. The stochastic variables are explained in the text.

is given as

2.4 Autoregressive hidden semi-Markov model
The integration of the models in Secs. 2.2 and 2.3 can be
described in terms of an extension of the HSMM. In one ofequivalent formulations [13] (also Sec. 3.3 of Ref. [11]), asemi-Markov model can be represented as a Markov modelon an extended state space. The extended state space isindexed by a pair (i, s) of the top state i (corresponding
to a musical event) and a counter of performed notes s =1, 2, . . . 2 with a transition probability
P (in , sn |in 1 , sn 1 ) =exit
sn ,1 P (in |in 1 )Pin 1 (sn 1 )⇣+ sn ,sn 1 +1 in ,in 1 1
where

Piexit(sn 1 )n 1

0
Piexit (s) = di (s)/⌃1
s0 =s di (s ).

⌘

(1)

(2)

Here in Eq. (1) denotes Kronecker’s delta. The exiting
probability in Eq. (2) represents the probability that theperformer moves to another event given that she has already played s notes at event i. The first term in the righthand side of Eq. (1) describes the probability that the performer moves to event in after having played sn 1 notes of
event in 1 . The second term describes the probability thatthe performer stays at event in and sounds another note
after having played sn 1 notes. In this way, this model describes the integrated process of performer’s progression
in the score and the production of performed notes.
The pitches and onset times of the performed notes
can be described with output probabilities associated withthis semi-Markov process. We assume the statistical independence of pitch and onset time for simplicity. The
output probability of pitch is given by P (pn |in , sn ) =Pipitch(pn ).
n
The output probability of the onset time of the n-th note
2 Remark: In the present model, s counts the number of notes played
during a musical event. This is not the durational time (in seconds) spenton that event, which is described with time interval IOI1.

P (tn |in , sn , in 1 , sn 1 , v, t1:n 1 )(w1 PIOI1 + w2 PIOI2 , sn = 1;=
PIOI3 ,
sn 6= 1

(3)

where
PIOI1 = PIOI1 (tn
PIOI2 = PIOI2 (tn
PIOI3 = PIOI3 (tntn s[n 1] |in , in 1 , v),(4)

tn 1 |in ) in in 1 .

(6)

tn 1 |in , in 1 , v),(5)(Here we have written s[n 1] = sn 1 to display the equation with clarity.) The three cases correspond to the threekinds of time intervals explained in Sec. 2.3. Because bothprobabilities for IOI1 and IOI2 have relevance in score
following, we have used a mixture probability of them
(w1 + w2 = 1). Such output probabilities with conditional dependence on the previous outputs have been considered in some studies on speech processing, and we callthe model autoregressive semi-Markov model based on theconvention of previous studies [12]. A graphical representation of the model is given in Fig. 2.
The distributions PIOI1 , PIOI2 , and PIOI3 can be estimated by analysing performance data. The functions PIOI2
and PIOI3 have previously been estimated with piano performance data [9]. It has been shown there that, in the mostimportant case that in = in 1 +1 (straight transition to thenext event), PIOI2 ( t|i+1, i, v) is well approximated by a
Cauchy distribution of the form
Cauchy( t; v(⌧iend⌧i )devi , 0.4 s).

(7)Here Cauchy(x; µ, ) denotes the Cauchy distribution
with median µ and width , and ⌧i is the onset score timeof event i, ⌧iend is the score time after which no new onsets
of event i can occur, and devi describes the ‘stolen time’of event i whose expectation value is given as the numberof short appoggiaturas and arpeggiated notes times the average IOI of the corresponding notes. Using this result, weProceedings of the 16th ISMIR Conference, Málaga, Spain, October 26-30, 2015
3.2 Comparison with the preprocessing method

can estimate PIOI1 in the case that in = in 1 +1 as
PIOI1 ( t|i+1, i, v) = Cauchy( t; v⌫i , 0.4 s)(8)where ⌫i = ⌧i+1 ⌧i is the note value of event i. The distribution PIOI3 was estimated with measurements on IOIs
of chordal notes and ornaments (see Secs. 3.3 and 4.2 of[9]).
Finally, tempo vn is estimated online with a separate model, for which we use a method based on switching Kalman filter (see Sec. 3.4 of [9]). In summary thecomplete-data probability P (i1:n , s1:n , t1:n , p1:n ) is given
as the following recursive product:
n h
Y

m=1

P (tm |im , sm , im 1 , sm 1 , vm 1 , t1:m 1 ) ·i
P (im , sm |im 1 , sm 1 )Pipitch(pm ) .
m395

To solve the problems with ornaments for score following, a preprocessing method has been proposed long ago[14]. The idea is to preprocess performed notes so thatornamental notes are not sent to the matching module directly. While the method can work for scores with notheavy polyphonic ornamentation and performances with
infrequent errors, the preprocessing can fail when there are
errors or unexpected repeats or skips near ornaments. Because a direct comparison showed that the HMM outperformed the preprocessing method for piano performanceswith errors, repeats, and skips [9], we compare our model
only with the HMM in Sec. 4.
3.3 Computational cost(9)

3. COMPARISON WITH OTHER MODELS
3.1 Relation to the HMM-based model
So far the state-of-the-art method for symbolic score following is developed with a performance model based ona standard HMM [9]. The current model can be seen as
an extension of this performance model in two ways. Firstthe transition probability of the HMM is realised as a special case of the transition probability in Eq. (1) with exiting
probabilities Piexit (s) constant in s. Specifically, it is given
as the inverse of the expected number of performed notesin event i. As is well known, this constraint leads to a geometrically distributed di (s) with a peak at s = 1, which is a
bad approximation for a large chord or a long trill/tremolo.
The second difference is the structure of output probabilities for onset times. In the standard HMM, the Markovian condition is assumed on the output probability of onset times. Thus the model describes only time intervals
IOI2 and IOI3, and the probability distribution for IOI1 in
Eq. (3) is ignored. In other words, the IOI output probability of the HMM assumes w1 = 0 and w2 = 1 in that equation. This means that the total duration of a trill/tremolo or
an arpeggio is poorly captured with the HMM.
These differences have important effects when the models are applied to score following. For score following,
the pitch information is generically most important. When
there are musical events with similar pitch contents in succession, however, the information on onset times and thenumber of performed notes play more significant roles incorrectly matching notes. For example, to correctly match
performed notes of succeeding trills/tremolos, the numberof notes and the duration of each trill/tremolo are important viewpoints. Since they are not well captured in the
HMM, the autoregressive HSMM would work better in thiscase. Similar situations arise for successions of arpeggios,
where the time intervals IOI2 and IOI3 are largely variable among realisations. On the other hand, the time intervals IOI1 and IOI2 are almost same for successive normal
chords and these IOIs carry much information necessary tocluster them. Thus the models are expected to have similareffects for passages without ornaments.

For score following, we find the most probable hidden state
sequence given the input performance. In order to realise
real-time processing, the computational cost of the estimation algorithm must be sufficiently small. We here compare
the present model and the HMM discussed in Sec. 3.1 interms of the computational cost.
The Viterbi algorithm can be applied for HMMs to estimate states. Let us denote the product of the transition probability and the output probability as aij (o) =P (j|i) · P (o|i, j) where o represents pitch and onset time.
The Viterbi update equation can be expressed as the following recursive equationp̂N (iN ) ⌘max

i1 ,...,iN

1⇥= max p̂N
iN

1

Y
N

ain 1 in (on )

n=1

1 (iN

1 )aiN

1 iN⇤(oN ) .

(10)
(11)

The number of states is N since a state corresponds to amusical event in the score. If we allow arbitrary progressions in the score including repeats and skips, a direct application of the Viterbi algorithm requires O(N 2 ) computations of probability for each update. When the probability matrix aij (o) can be represented as a sum of a band
matrix ↵ij of width D and an outer product of two vectors Si and rj , the computational complexity can be reduced to O(DN ) with a recombination method [7]. Intuitively, ↵ij describes probabilities corresponding to transitions between neighbouring states, which have larger probabilities, and Si and rj represent probabilities corresponding to large repeats and skips, which typically have verysmall probabilities. Substituting aij (o) = ↵ij + Si rj
into Eq. (11), we see ↵ij induces O(DN ) complexity andSi rj induces O(N ) complexity by a recombination. This
simplified transition probability matrix is used in previous
studies to enable real-time processing for long scores.
It is clear from the formulation of the autoregressive
HSMM in Sec. 2.4 that the standard Viterbi algorithm canalso be applied to the model. In practice, we put an upper bound on the number of performed notes smaxforieach event i, and the number of states of the HSMM is⌃i smax⌘ SN where S is the average of smax
. Becauseiiof the special form of transition probabilities in Eq. (1), the396

Proceedings of the 16th ISMIR Conference, Málaga, Spain, October 26-30, 2015

Table 1. Error rates (%) of score following with the autoregressive HSMM (“HSMM”), the hybrid model (“Hybrid”), and the HMM [9]. The first four pieces indicate
Couperin’s Allemande à deux clavecins, the solo piano partof Beethoven’s first piano concerto, Beethoven’s second
piano concerto, and Chopin’s second piano concerto [9],and the last two pieces are explained in the text.
Piece
Couperin
Beethoven 1
Beethoven 2
ChopinDebussy
Tchaikovsky# Notes
1763
17587
5861
16241
3294
2245

HSMM
5.50
3.16
2.01
9.22
3.64
0.40

Hybrid
6.02
3.13
2.20
9.22
3.58
0.40

HMM
6.66
3.16
2.35
11.1
4.66
4.55

computational complexity for one Viterbi update is generically O(SN 2 ). When we apply the recombination method
in Ref. [7], the complexity can be reduced to O(DSN )for the outer-product type transition probability. Note thatthe width D in the top-level transition probability matrix
induces SD transitions between HSMM states. Consequently the computational cost of the model is about S
times larger than its reduced HMM. For example, if we
set smaxas twice the number of expected notes per event,i
S ' 3–10 for a score with a modest degree of polyphony,and it increases if there are many large chords or longtrills/tremolos.
3.4 Hidden hybrid Markov/semi-Markov model
As discussed in Sec. 3.1, there are reasons that the present
model yields better results for score following than the
HMM, but it is at the cost of increased computational cost,which is unwanted for long scores. On the other hand,most of the musical events in scores are normal chords(or single notes) for which the HMM already yields goodresults. Therefore if we combine the HMM state representation for normal chords and the autoregressive HSMM
state representation for other ornamented events, it would
be possible to obtain an improved score-following algorithm with minimal increase in computational cost. Such a
combination of HMM and HSMM can be achieved in theframework of hidden hybrid Markov/semi-Markov model[5, 15]. In the hybrid model, normal chords are represented with HMM states and other events (i.e. trill,
tremolo, arpeggio, short appoggiatura, and after notes) arerepresented with HSMM states. For this model the computational complexity of the Viterbi algorithm takes thesame form as the autoregressive HSMM, by substituting
smax= 1 for HMM states in S = ⌃i smax
/N .
ii
4. COMPARING THE ACCURACY OF SCOREFOLLOWING
To evaluate and compare the discussed models with respect to the accuracy of score following, we implemented

Table 2. Number of mismatched notes of various types.
Each type is explained in the text. The same abbreviations
for the models as in Table 1 are used.
Type
Trill
Tremolo
Arpeggio
Other
ornaments
Other# Notes
8159
2603
1081

HSMM
282
115
36

Hybrid
281
115
33

HMM
508
151
127

2401

340

339

362

32030

1580

1599

1673

three score-following algorithms based on the autoregressive HSMM (Sec. 2.4), the hybrid model (Sec. 3.4), and theHMM [9], and run these algorithms for music performancedata containing various ornaments. In addition to the piano
performance data used in Ref. [9] which contains performance errors, repeats and skips, we used collected piano
performances of passages in Debussy’s En Blanc et Noirwith successions of tremolos (the first piano part in the second movement) and the solo piano part of Tchaikovsky’s
first piano concerto with his typical successions of wide
arpeggios (the last section of the second movement).
The additional parameters i for the autoregressive
HSMM and the hybrid model were set as follows: i =0.4speakfor trills and tremolos and i = 1 otherwise. Theimixture weights for the output probability for time intervals IOI1 and IOI2 were set as w1 = w2 = 1/2. These
parameters were used as a benchmark and there is a roomfor further optimisation.
For the evaluation measure, we calculated the error rate,
which is defined as the proportion of mis-matched notes
to the total number of performed notes. There were performed notes that are difficult to associate with any scorenotes even for humans, which naturally appear in real data.
While they were included in the input data, they were notused in the calculation of error rates. Results are shown in
Table 1, where we see that the autoregressive HSMM andthe hybrid model had similar accuracies, and the HMM had
the worst accuracy overall. (Slight differences in the values
for the HMM compared to those in Ref. [9] are mainly due
to slight corrections of the implementation.) For detailed
error analysis, we list the frequencies of classified matching errors in Table 2. Here the numbers indicate the total
number of matching errors in the whole data for each type.
Ornaments are classified into the first four types, and othernotes are gathered in the last type. Significant reductionof matching errors is observed in the first three types (trill,
tremolo, and arpeggio), and other types of matching errors
are also reduced but rather slightly in the reduction rate.
Two example results of score following are shown in
Fig. 3, which represent typical situations where the autoregressive HSMM worked better than the HMM. In thefirst example, the passage includes a succession of tremolos with similar pitch contents. We see some of the mismatched notes with the HMM are correctly matched withthe autoregressive HSMM. Similarly the mismatched notes

Proceedings of the 16th ISMIR Conference, Málaga, Spain, October 26-30, 2015

397

Table 3. Averaged computation time (ms) required for one
Viterbi update. The same abbreviations for the models andthe musical pieces as in Table 1 are used.
Piece
Couperin
Beethoven 1
Beethoven 2
ChopinDebussy
Tchaikovsky

Score

Performance

HSMM
1.6
5.9
7.0
7.1
0.9
1.2

Hybrid
1.1
2.9
3.0
3.5
0.8
1.0

HMM
0.3
1.1
1.6
1.2
0.1
0.1

(a) A passage from Debussy’s En Blanc et Noir with the autoregressive
HSMM.
Score

Performance(b) Same as (a) with the HMM.
Score

with the HMM are all correctly matched with the autoregressive HSMM for a succession of wide arpeggios in thesecond example. These results are consistent with the discussion in Sec. 3.1.
We also measured the required computation time (Table 3). The computation time for each Viterbi update isconstant over time, and the algorithms were run on a laptop with moderate computation power. The results confirm our expectation that the use of hybrid model for score
following has practical advantages over the autoregressive
HSMM in the computation time and the HMM in the accuracy.

Performance

5. CONCLUSION(c) A passage from Tchaikovsky’s first piano concerto with the autoregressive HSMM.
Score

Performance(d) Same as (c) with the HMM.

Figure 3. Example results of score following with theautoregressive HSMM and the HMM [9]. Mismatchednotes are indicated with bold red lines.

We explained reasons that the present model of symbolic music performance based on autoregressive HSMM
is more advantageous for score following than previously
studied HMMs, and we have confirmed this empirically bycomparing the accuracy of score following and analysing
the matching errors. Because a semi-Markov model canbe seen as a Markov model with an extended state space as
we have explained, we can apply to the present model themethods for HMMs to improve score following [7, 16]. In
particular, this is important to reduce matching errors occurring after repeats and skips and those due to reordered
notes in the performance, which were the main factors ofremaining errors.
It would be interesting to apply the present model formusic/rhythm transcription and related problems. Because
the model describes both the total duration and the internal
temporal structure of ornaments, it would be possible todetect ornaments from performances without a score andintegrate the results into music transcription.

6. ACKNOWLEDGEMENTS
This work is partially supported by NII MOU Grant
in fiscal year 2014 and Grant-in-Aid for Scientific Research from Japan Society for the Promotion of Science,
No. 26240025 (S.S. and N.O.) and No. 25880029 (E.N.).

398

Proceedings of the 16th ISMIR Conference, Málaga, Spain, October 26-30, 2015
7. REFERENCES[1] R. Dannenberg, “An on-line algorithm for real-time accompaniment,” Proc. ICMC, pp. 193–198, 1984.
[2] B. Vercoe, “The synthetic performer in the context oflive performance,” Proc. ICMC, pp. 199–200, 1984.
[3] N. Orio, S. Lemouton and D. Schwarz, “Score following: State of the art and new developments,”
Proc. NIME, pp. 36–41, 2003.
[4] B. Pardo and W. Birmingham, “Modeling form for online following of musical performances,” Proc. of the20th National Conf. on Artificial Intelligence, 2005.
[5] A. Cont, “A coupled duration-focused architecture forrealtime music to score alignment,” IEEE Trans. PAMI,
32(6), pp. 974–987, 2010.
[6] A. Arzt, G. Widmer and S. Dixon, “Adaptive distancenormalization for real-time music tracking,” Proc. EUSIPCO, pp. 2689–2693, 2012.
[7] E. Nakamura, T. Nakamura, Y. Saito, N. Ono and
S. Sagayama, “Outer-product hidden Markov modeland polyphonic MIDI score following,” JNMR, 43(2),pp. 183–201, 2014.
[8] P. Cuvillier and A. Cont, “Coherent time modelingof semi-Markov models with application to real-time
audio-to-score alignment,” Proc. IEEE MLSP, 6 pages,
2014.
[9] E. Nakamura, N. Ono, S. Sagayama and K. Watanabe,“A stochastic temporal model of polyphonic MIDI performance with ornaments,” to appear in JNMR, 2015.
[10] N. Orio and F. Déchelle, “Score following using spectral analysis and hidden Markov models,” Proc. ICMC,
pp. 1708–1710, 2001.
[11] S.-Z. Yu, “Hidden semi-Markov models,” Artificial Intelligence, 174, pp. 215–243, 2010.
[12] J. Bilmes, “Graphical models and automatic speech
recognition,” in Mathematical foundations of speech
and language processing (Springer New York),
pp. 191–245, 2004.
[13] M. Russel and A. Cook, “Experimental evaluation ofduration modelling techniques for automatic speech
recognition,” Proc. ICASSP, pp. 2376–2379, 1987.
[14] R. Dannenberg and H. Mukaino, “New techniques forenhanced quality of computer accompaniment,” Proc.
ICMC, pp. 243–249, 1988.
[15] Y. Guédon, “Hidden Hybrid Markov/Semi-Markov
Chains,” Computational Statistics and Data Analysis,
49, pp. 663–688, 2005.
[16] E. Nakamura, Y. Saito, N. Ono and S. Sagayama,“Merged-output hidden Markov model for score following of MIDI performance with ornaments, desynchronized voices, repeats and skips,” Proc. JointICMC|SMC 2014, pp.1185-1192, 2014.