POLYPHONIC AUDIO-TO-SCORE ALIGNMENT BASED ON BAYESIAN LATENTHARMONIC ALLOCATION HIDDEN MARKOV MODEL
Akira Maezawa,† Hiroshi G. Okuno,† Tetsuya Ogata,† Masataka Goto‡
Dept. of Intelligence Science and Technology ‡ National Institute of Advanced Industrial
Graduate School of Informatics, Kyoto University
Science and Technology (AIST)
Sakyo, Kyoto 606-8501 Japan
Tsukuba, Ibaraki 305-8568 Japan†

ABSTRACTThis paper presents a Bayesian method for temporally aligning a
music score and an audio rendition. A critical problem in audio-toscore alignment is in dealing with the wide variety of timbre andvolume of the audio rendition. In contrast with existing works that
achieve this through ad-hoc feature design or careful training of tonemodels, we propose a Bayesian audio-to-score alignment method bymodeling music performance as a Bayesian Hidden Markov Model,
each state of which emits a Bayesian signal model based on Latent
Harmonic Allocation. After attenuating reverberation, variational
Bayes method is used to iteratively adapt the alignment, instrument
tone model and the volume balance at each position of the score. The
method is evaluated using sixty works of classical music of a variety
of instrumentation ranging from solo piano to full orchestra. We
verify that our method improves the alignment accuracy compared to
dynamic time warping based on chroma vector for orchestral music,
or our method employed in a maximum likelihood setting.
Index Terms— Audio-to-score alignment, Variational Bayes inference
1. INTRODUCTION
Audio-to-score alignment, the task of finding a temporal mapping
between a music score and audio signal, is a critical task in music
information retrieval. It is required whenever a system needs to coordinate a music score (e.g. a standard MIDI file) and an audio signal,such as score-informed source separation [1, 2], or score-informed
analysis of music performance [3, 4, 5, 6, 7].
Audio-to-score alignment is difficult because the music scoreconveys only the sequence of instrument and pitch in which a pieceshould be played – very little on the fluctuation of tempo, timbre andbalance of dynamics are conveyed. For example, an orchestra mightemphasize a particular instrument part more than the other compared to another orchestra. One orchestra might have a brilliantlysounding brass section compared to another. One orchestra mightslow down a particular section of music, while another orchestra
might speed it up a bit. Audio-to-score alignment should absorb
such discrepancies in dynamics timbre and tempo.
Most approaches are based on designing timbre-robust feature
for alignment, using ad-hoc distance measure. We found that, while
there are basic tenets of designing features and distance measures, a
big portion of it is still an art that requires careful adjustments. A
critical problem is that of removing timbral dependence; aligning apiece of music that is rich in timbre, such as a symphony, is difficult.
We believe that the problem of spectral mismatch could be alleviated only to an extent through design of timbre-robust features.
To cope with timbre, we should instead consider timbre as an unknown quantity a priori, that we infer from the observed musicalaudio. Such philosophy calls for a Bayesian treatment of timbre inaudio-to-score alignment.

978-1-4577-0539-7/11/$26.00 ©2011 IEEE

185

In this paper, we propose an audio-to-score alignment methodby fitting a Bayesian Hidden Markov Model (HMM) based on Latent
Harmonic Allocation (LHA) signal model [8]. LHA is a Bayesiantreatment of musical signal based on fitting a superposition of histograms onto the observed power spectrum. Our method is basedon three assumptions. First, we assume that the volume of eachnote is different, but remains more-or-less consistent within a note
(intra-note volume consistency). Second, the timbre of each pitch /instrument pair is unique, but remains consistent throughout a piece(intra-music timbre consistency). Third, each position of the musicscore emits a combination of pitch / instrument pairs and a stationary
noise. We evaluate the effectiveness of treating timbre and volume
as random quantities using sixty pieces of classical music.
2. EXISTING WORKS
Audio-to-score of a piece with a wide range of timbre has been
attacked mainly using two approaches to feature selection. First
approach uses features that are robust to timbral differences. For
example, many studies use chroma vector, a timbre-robust featurethat reflects the power in a particular pitch-class [9, 10, 11, 12, 13].
Other studies use features that are reminiscent of the chroma vector, but with improved robustness against timbral variety [14, 15].
Second approach is based on a generative model of spectrograms[16, 17]. For temporal matching, Dynamic Time Warping (DTW),
HMM, or Dynamic Bayesian Model (DBM) have been employed,though DTW is used, by far, most frequently.
3. BAYESIAN AUDIO-TO-SCORE ALIGNMENT BASEDON LHA-HMM
Our method estimates audio-to-score alignment by modeling the musical audio signal as a Bayesian HMM, each state of which emits asignal based on LHA model.
3.1. Preprocessing
Given an input signal o(t), its short-time Fourier Transform (STFT),
O(f, t), of size F -by-T is evaluated. Then, reverberation is attenuated by modeling O(f, t) as an auto-regressive process with X(f, t)
as the source signal. This model was inspired from that used inspeech dereverberation [18]. Each frequency bin contains linear predictive coefficients G(f, i) of order P. In other words, we model the
dereverberated STFT X(f, t) as follows:PO(f, t − i)G(f, i)(1)
O(f, t) = X(f, t) +i

G(f, i) is estimated using linear prediction.
3.2. Latent Harmonic Allocation
LHA is a generative signal model proposed by Yoshii, which interprets the spectrogram as a histogram, each bin of which is considered

ICASSP 2011to be multiple draws from various musical instrument [8]. The probability of observing the energy from a particular musical instrument
follows a Normal distribution centered around integer multiple ofits fundamental frequency. The probability of drawing a particular
overtone of a particular instrument, both of which follow multinomial distribution, have a conjugate prior. Variational expectationmaximization is used for inference.
3.3. Joint Estimation of Alignment, Timbre, Pitch and Volume
We assume that power contained in each time-frequency bin ofX(f, t) originated from some combination of power originatingfrom different musical instruments playing at different pitch. Hence,the source signal X(f, t) is modeled as a sum of music instrument
histograms. Assuming that a particular music contains I unique
instrument/pitch pairs, each with the fundamental frequency distributed normally with mean μi and precision λi , the model fits asequence of a combination of these pairs onto X(f, t).
To attain intra-music timbre consistency for each of I instrument/pitch pairs, each instrument/pitch pair has a unique tonemodel, consisting of the fundamental frequency and the relative
strengths of the overtones. This tone-model is shared for all occurrences of the given instrument/pitch pair in a piece. Our model models intra-note volume consistency by incorporating a I-dimensionalnote emission vector, ith element of which indicates the likelihoodthat ith instrument/pitch pair contributes to the observation.
The model joins a sequence of D note-emission vectors by a leftto-right HMM, where D is the number of states of the music score.
The emission vector associated with the dth state is set such that it
is highly likely to emit pitch/instrument pairs that are notated at thatpoint in the music score, and highly unlike to emit others. sd (t) is abinary vector, which is 1 if state at time t is d, and 0 otherwise.
To infer the alignment, we introduce various latent variables.
Y = {Y (f, i)} is a random variable of size I × F , each of which isa J-dimensional multinomial variable jth element of which governs
the ratio of power the jth overtone contributes in frequency f for
ith instrument/pitch pair. Furthermore, Z = {Z(f, d)} is a randomvariable of size F ×D, each of which is a I-dimensional multinomial
variable, ith element of which governs the ratio of power the the ith
instrument/pitch pair contributes in frequency f .
The complete data likelihood is given as follows:× p(Y |Z, S, A)p(S|τ )p(E)p(A)p(τ )p(μ, λ) (2)
where
 
X(f,t)Zi (f,d)Yj (f,i)Sd (t)p(X|Y, Z, μ, λ, S) =N f |jμi , j 2 λ−1it,i,j,f,dei (d)X(f,t)Zi (f,d)Sd (t)t,i,f,dp(Y |A, Z, S) =aj (i)X(f,t)Zi (f,d)Yj (f,i)Sd (t)t,i,j,f,dsp(μ, λ; ν, b, m) =

N G(μk , λk ; mk , bk , lk , νk )kp(τ ) =

Dir(τ (j); τ0 (j))

p(S|τ ) =jp(E; ) =

D



t

Dir(e(d); d )p(A; α) =d f +1/2
N (f ) ≈ f −1/2 Nc (f )df˜ = Nc (f ). N G is a Normal-Gamma
distribution, whose probability density is given as follows:
N G(μ, λ|m, b, l, ν) =

√

l−1/2
b
ν l λk1√
exp − bλ(μ − m)2 − λν
(3)2
Γ(l) 2π
where Γ(i) is the Gamma function. Dir(a) is the dirichlet distribution. The complete graphical model is shown in Fig. 1.
We seek to maximize the posterior p(S, Z, Y, E, A, μ, λ, τ |X),
but this is analytically intractable. Therefore, we instead maximize
an approximation of the posterior, q(S, Z, Y, E, A, μ, λ, τ ). We assume the approximation is factorized into the following form:
q(S, Z, Y, E, A, μ, λ, τ ) =qS (S)qZ (Z)qY (Y )qE (E)qA (A)qμ,λ (μ, λ)qτ (τ ) (4)We optimize the approximate posterior by minimizing the
Kullback-Leibler divergence between it and the posterior. Such variational minimization yields in the following update [19]:log qZi (Zi ) := c + log p(X, S, Z, Y · · · τ )¬Zi(5)¬Zi means all factors of q(S, Z, Y, · · · , τ ) excluding Zi , andf (x, y)y means to take the expectation of f (x, y) under q(y). The
iteration terminates when the following
quantity has converged:F(qx (x), θ) =qZ (Z) log p(X,Z)
dZqZ (Z)(6)

For convenience, we define the following quantities:p(X, Z, Y, S, E, A, μ, λ) = p(X|S, Y, Z, μ, λ)p(Z|E, S)

p(Z|E, S) =Fig. 1. Graphical model of our method.

τj (k)sj (t)sk (t−1)

j,kξj (f, i) = Yj (f, i)Y(7)

ηd (i) = sd (t)sγi (f, d) = Zi (f, d)Z

(8)

lAj (i) = log aj (i)E (11)(10)

lτd (d) = log τd (d)τ (9)lEi (d) = log ei (d)E (12)
lNf (i, j) = log N f |jμi , j 2 λ−1
(13)
i
μ,λ

ζi (t, j) = si (t)sj (t − 1)s
where

M
α
(i)
lAj (i) = ψ(αj (i)) − ψ
l
l=1

K

(d)
lEi (d) = ψ(i (d)) − ψ
l
l=1

 
2
1 li f
1− log 2πνi + ψ(li )lNf (i, j) = −− mi +
2 νi j
bi

(14)
(15)
(16)
(17)

Here, ψ(x) is the digamma function.

Dir(a(k); α(k))

3.4. Updating Note Emissionk

N (f |μ, λ−1 ) is a discrete distribution over f ∈ Z, in which: f +1/2
N (f |μ, λ−1 ) =Nc (f˜|μ, λ−1 )df˜
f −1/2where Nc is a Normal distribution. We approximate this N by

186

First we update the responsibility ith pitch/instrument pair has at agiven frequency f , for each state of the HMM d to the following:ρi (f, d)qZ (Z) =γi (f, d)Zi (f,d) where γi (f, d) =(18)i ρi (f, d)i,f,dwhere ρ is the following:X(f, t)ηd (t)log ρi (f, d) =
t





ξj (f, i) lNf (i, j) + lAj (i)× lEi (d) +

(19)

j

Furthermore, the note-emission vector is updated as follows:
qE (E) =Dir ed ; i (:) +η: (t)X(f, t)γi,j (f, :)
i

(20)

t,f,j

3.5. Updating Tone-model

Note that this has the same functional form as a HMM, with transition probability replaced by exponent of log-expectation, and theemission probability replaced κd (f ). Hence, κd (f ) can be considered as a subnormalized histogram that represents the spectrum thedth state is likely to emit. To compute the expectation of the sufficient statistics η and ξ, forward-backward algorithm is used. Forward probability α and backward probability β are given as the following recursions:α(t, d) = p(s(t)|X(1) · · · X(t))

1 α(t − 1, d )elτd (d)κd (f )X(f,t)=
Z First, we update the responsibility jth overtone has at a given frequency f , for each note / instrument pair i:φj (f, i)qY (Y ) =ξj (i, f )Yj (f,i) where ξj (i, j) =(21)
k φk (f, i)j,i,f

where φ the following:log φj (f, i) = lNf (i, j) + log aj (i)X(f, t)ηd (t)γi (f, d)×(22)β(t, d) = p(Xt+1 (f ) · · · XT (f )|sd (t) = 1)β(t + 1, d )κd (f )X(f,t+1) elτd (d )
=
d

(34)

f

d

(35)

f

These are used to arrive at the sufficient statistics:
1sj (t)s = α(t, j)β(t, j)(36)Zα(t − 1, j)β(t, k) κk (f )X(f,t) elτj (k) (37)sj (t − 1)sk (t)s =
Z
f

t,d

The fundamental frequency and its variance is set to the following:N G(μk , λk ; mk , bk , lk , νk )(23)
qμ,λ (μ, λ) =4. EXPERIMENT

We evaluate the performance of our system against a widely-usedalignment method. Moreover, we evaluate the significance of performing audio-to-score alignment in a Bayesian setting as opposed
mk bk + Nγ,k x/jψkto maximum likelihood estimation, and the effectiveness of adapting
(24)
mk :=
bk + Nγ,kthe volume and timbre information for aligning music.
Our method is tested against three conditions. First con(25)bk := bk + Nγ,kdition is alignment based on Dynamic Time-warping (DTW) of(26)
lk := lk + Nγ,kchroma-vector, perhaps the most popular method for audio-to-score2alignment. Second, alignment based on non-adaptive, maximum1 bk Nγ,kx/jψ(k) − mk
νk := νk +likelihood LHA-HMM is used as a baseline for non-Bayesian frame2 bk + Nγ,kwork of our method. Essentially, Viterbi decoding is done on a
2
2
HMM which has τ qτ (τ ) , instead of exp log τ qτ (τ ) for statex/j − x/jψ(k)+ Nγ,k(27)
ψ(k)
transition probability. Furthermore, κ is replaced with a fixedand normalized spectral histogram κd (f ) =where ψk (f, j) is the following multinomial distribution:i,j i (d) αj (i)

,
l
/(b
v
))
;
this
signalmodel
issimilarto one used in
N
(f
|jμ
i
k
k k
ψf,j (k) =γk (f, d)ξj (f, k)η( d, t)X(f, t)/Nγ,k(28)[16]. Finally, LHA-HMM without volume/timbre adaptation is usedd,t
as a baseline for Bayesian LHA-HMM without timbre update.
andFor evaluation, we align sixty pieces from RWC Classical Muγk (f, d)ξj (f, k)η( d, t)X(f, t)(29)
Nγ,k =
sic
Database[20], and compare them against the database’s beat dataf,j d,t[21]. To evaluate our method for classical music with various instruThe relative strengthof
each
overtoneis
updatedtothefollowing:⎛⎞mentations, we divide the database into five categories: orchestral(C001 to C010), non-orchestral ensemble (C012 to C021), solo pi⎝⎠(30)qA (A)= Dir a(i); α(i)+ X(f, t)γi (f, d)ηd (t)ξ(f, i)ano (C022 to C035), solo instrumental/duo (C036-C011, C011), andit,f,dvocal music (C045 to C050). C005 is omitted because it includes a
cadenza, a freely-played section that is not notated on the score.
All signals are downsampled to 8kHz and analyzed using Han3.6. Updating HMM
ning window of length 2048 with overlap of 400 samples. One perThe state transition probability
sistent tone with fundamental of 500Hz and variance of each Gaus τ is updated as follows: sian set to 500Hz was used as the noise model. The prior of note
Dir τ (k);ζ(t, k) + τ0 (k)(31)
qτ (τ ) =emission are set such that, all unnotated notes have hyperparametertk
set to 10−6 , and all notated notes and noise set to 1 (non-informative
The probability of state sequence is given as follows:
prior). For each tone model, mk was set to the notated fundamental
frequency, and the variance to 50Hz (lk = 106 , νk = 50 × 106 ,
T 



S (t)6d (t−1)κd (f )X(f,t) d(32) bk = 50 × 10 ). Moreover, we consider up to the tenth overtone
log τj (i)S
qS (S) =
τ
(J
=
10)
for
each
tone-model, and the J-dimensional timbre model
t=1 d,dfαj (k) set to non-informative.
where
To optimize the posterior, we iteratively update γ, ξ, τ and η
log κd (f ) =until convergence, after which A is updated. This is repeated until convergence. After convergence E was updated. μ and λ were
γi (f, d) lEi (d) +ξj (f, i) lNf (i, j) + lAj (i)(33)
not updated, as we assumed the shape of harmonics do not changeijsignificantly within a piece of music. The result is shown in Table 1.
with

k

187

Table 1. Percentage of estimated alignment whose error  is withina given margin. “Type” indicates the category of pieces, where OR
is orchestral, EN is non-orchestral ensemble, PN is piano-solo, IN
is non-piano solo/duo, and VO is vocal music. Four conditions,
Chroma-based DTW, Maximum Likelihood of our method, ourmethod without timbre and volume update, and proposed method,
are labeled “C,” “ML,” “N,” and “P,” respectively.
Type <50ms  <.1s  <.5s  < 1s
OR
C
8.0%
16.3%
57.7%
70.3%
1.2%
1.8%
4.1%
6.5%
OR
ML
24.4%
41.4%
65.1%
72.4%
OR
N
OR
P
28.2% 46.8% 70.8% 75.0%
EN
C
14.7%
29.1%
69.5% 83.4%
0.1%
0.1%
0.5%
1.1%
EN
ML
16.8% 36.0%
71.7%
80.7%
EN
N
EN
P
16.2%
35.5% 72.0%
80.9%
PN
C
12.3% 26.4%
56.3% 65.4%
0.1%
0.1%
0.5%
1.0%
PN
ML
PN
N
6.7%
18.8%
56.3%
63.3%
PN
P
6.6%
18.8% 56.9%
63.7%
IN
C
11.8%
25.1% 71.1% 81.8%
1.7%
2.6%
3.4%
4.1%
IN
ML
IN
N
24.3%
40.4%
67.2%
73.2%
24.8% 40.7%
67.5%
73.4%
IN
P
VO
C
6.2%
11.9%
39.1% 54.3%
VO
ML
0.1%
0.1%
0.2%
0.2%
VO
N
8.3% 14.4% 40.2%
49.5%
VO
P
7.9%
13.9%
39.6%
49.2%
5. DISCUSSION
Our method is capable of generating alignment whose quality iscomparable to that of well-established chroma-based DTW alignment. This is significant because DTW based on timbre-robust features is still an art that requires finesse in design of features anddissimilarity measure. On the other hand, the only ad-hoccery employed in our method was the choice of the magnitude of noteemission vector for un-notated notes, and the initial variance of each
Normal-Gamma distribution. Otherwise, our method was initialized with non-informative priors. Treating timbre and volume as
fixed quantity (“ML”) significantly degrades the alignment, suggesting that it is in the Bayesian treatment of timbre and volume that
achieves the performance of our method.
The performance of our model tends to degrade slightly when
we update the timbre and volume, as seen in the results of “N” and“P.” This is because posterior of a Dirichlet (i.e. timbre and noteemission) has smaller variance. Hence, updating timbre and volume
model makes our model more selective to musical signals that satisfy
intra-note volume consistency and intra-music timbre consistency.
In reality, these assumptions do not strictly hold, meaning that it isbetter to deal with timbre and note emission as Dirichlet with a large
variance instead of coercing it into a particular multinomial. It is
important to note that both “N” and “P” are guaranteed to converge;
they are two different ways to align using LHA-HMM.
Our method performs poorly in piano solo and vocal music, as
seen from the low percentage of errors that lie within 50ms. This isbecause they violate intra-note volume consistency and intra-music
timbral consistency, respectively. Intra-note volume consistency isviolated in piano music because the volume of piano decays withina note. Intra-music timbre consistency is violated in vocal music
because a singer can produce a variety of phonemes.
On the other hand, the method is effective for orchestral pieces.
This is because orchestral music is rich in timbral content and dynamic range; achieving timbre robustness is hard in this situation,but our model absorbs differences in timbre and dynamics.

188

6. CONCLUSION
This paper presented a Bayesian approach to audio-to-score alignment using LHA model and HMM. Our method, unlike most existing methods using DTW, does not require careful adjustments ofmodel parameters and skillful design of timbre-robust features anddissimilarity measures. Yet, our method performed similarly to thatof DTW-based method using ad-hoc timbre-robust features.
In the future, we plan to incorporate more sophisticated modelfor note duration. Moreover, we plan to explore other models for
timbre and dynamics, to better deal with piano and vocal music.
Acknowledgment The authors thank Kazuyoshi Yoshii for
helpful discussions on Latent Harmonic Allocation inference. This
work was funded by Kakenhi(S) and GCOE.
7. REFERENCES[1] K. Itoyama et al., “Integration and Adaptation of Harmonic andInharmonic Models for Separating Polyphonic Musical Signals,” in ICASSP, Apr. 2007, pp. I–57–I–60.
[2] Y. Han and C. Raphael, “Desoloing Monaural Audio Using
Mixture Models,” in ISMIR, 2007, pp. 145–148.
[3] A. Maezawa et al., “Query-By-Conducting: An Interface toretrieve classical-music interpretations by real-time tempo input,” in ISMIR, 2010, pp. 477–482.
[4] A. Maezawa et al., “Bowed String Sequence Estimation ofa Violin Based on Adaptive Audio Signal Classification andContext-Dependent Error Correction,” in ISM, 2009, pp. 9–16.
[5] A. Maezawa et al., “Violin Fingering Estimation Based on Violin Pedagogical Fingering Model Constrained by Bowed Sequence Estimation from Audio,” in IEA/AIE, 2010.
[6] M. Molina-solana et al., “Evidence for pianist-specific rubato
style in chopin nocturnes,” in ISMIR, 2010, pp. 225–230.
[7] C. S. Sapp, “Comparative analysis of multiple musical performances,” in ISMIR, 2007, pp. 2–5.
[8] K. Yoshii and M. Goto, “Infinite Latent Harmonic Allocation:
A nonparametric Bayesian approach to multipitch analysis,” in
ISMIR, 2010, pp. 309–314.
[9] C. Joder et al., “An improved hierarchical approach for musicto-symbolic score alignment,” in ISMIR, 2010, pp. 39–44.
[10] R. Macrae and S. Dixon, “Accurate real-time windowed time
warping,” in ISMIR, 2010, pp. 423–428.
[11] B. Niedermayer et al., “A multi-pass algorithm for accurate
audio-to-score alignment,” in ISMIR, 2010, pp. 417–422.
[12] N. Orio et al., “Score Following : State of the Art and New
Developments,” in NIME, 2003, pp. 36–41.
[13] N. Hu et al., “Polyphonic audio matching and alignment formusic retrieval,” in WASPAA, 2003, pp. 185–188.
[14] M. Muller and S. Ewert, “Towards Timbre-Invariant Audio
Features for Harmony-Based Music,” IEEE TASLP, vol. 18,
no. 3, pp. 649–662, Mar. 2010.
[15] M. Muller and F. Kurth, “Enhancing similarity matrices formusic audio analysis,” in ICASSP, 2006, pp. 9–12.
[16] C. Raphael, “A Hybrid Graphical Model for Aligning Polyphonic Audio with Musical Scores,” in ISMIR, 2004, pp. 387–
394.
[17] A.T. Peeling, P. Cemgil and S. Godsill, “A Probabilistic
Framework for Matching Music Representations,” in ISMIR,
2007, pp. 267–272.
[18] T. Yoshioka et al., “Dereverberation by using time-variant nature of speech production system,” EURASIP J. Adv. Signal
Process, vol. 2007, no. 2, pp. 6–6, 2007.
[19] M. J. Beal, Variational Algorithms for Approximate BayesianInference, Ph.D. thesis, University College London, 2003.
[20] M. Goto, “Development of the RWC Music Database,” in Int’lCongress on Acoustics, 2004, vol. I, pp. 553–556.
[21] M. Goto, “AIST Annotation for the RWC Music Database,” in
ISMIR, 2006, pp. 359–360.