STATISTICAL MUSIC STRUCTURE ANALYSIS BASED ON A
HOMOGENEITY-, REPETITIVENESS-, AND REGULARITY-AWARE
HIERARCHICAL HIDDEN SEMI-MARKOV MODEL
Go Shibata
Ryo Nishikimi
Eita Nakamura
Kazuyoshi Yoshii
Graduate School of Informatics, Kyoto University, Japan{gshibata, nishikimi, enakamura, yoshii}@sap.ist.i.kyoto-u.ac.jp

ABSTRACTThis paper describes a music structure analysis method thatsplits music audio signals into meaningful segments suchas musical sections and clusters them. In this task, how
to model the four fundamental aspects of musical sections,
i.e., homogeneity, repetitiveness, novelty, and regularity,
in a unified way is still an open problem. Here we propose a solid statistical approach based on a homogeneity, repetitiveness-, and regularity-aware hierarchical hidden
semi-Markov model. The higher-level semi-Markov chain
represents a sequence of sections that tend to have regularly spaced boundaries. The timbral features in eachsection are assumed to follow emission distributions thatare homogeneous over time. The lower-level left-to-rightMarkov chain in each section represents a chord sequence
whose sequential order is constrained to be a repetition ofa chord sequence in another section of the same cluster.
The whole model can be trained unsupervisedly based on
Bayesian sparse learning where unnecessary sections automatically degenerate. The proposed method outperformed
representative methods in segmentation and clustering accuracies with estimated sections having similar statisticalproperties as the ground truth data.
1. INTRODUCTION
Music structure analysis is a long-standing research topic[1] because detection of meaningful segments called musical sections (e.g., intro, verse, bridge, and chorus sectionsin popular music) from music audio signals forms a basis of music information retrieval (MIR). In general, music
structure analysis involves a segmentation step that splits
music signals into sections [2–9], a clustering step that categorizes such sections into several classes [10–18], and alabeling step that gives each section a concrete label suchas “verse A”, “verse B”, or “chorus” [19–21]. We here
tackle the segmentation and clustering for popular music.
In previous studies, sections of popular music havebeen characterized in three aspects, i.e., homogeneity referc G. Shibata, R. Nishikimi, E. Nakamura, and K. Yoshii.
Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Attribution: G. Shibata, R. Nishikimi, E. Nakamura, and K. Yoshii. “Statistical Music Structure Analysis Based on a
Homogeneity-, Repetitiveness-, and Regularity-Aware Hierarchical Hidden Semi-Markov Model”, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.

268

Regularity of section durations
Section A
C

GSection B

Am

Em

F

C

F

G
Vocal

Homogeneity of
timbral features

Repetitiveness ofchord progressions

GuitarSection C

Section A
CGGuitar

Am

Em

F

C

F

G
Vocal

GuitarSax

GuitarFigure 1. Music structure analysis based on homogeneityof timbral features, repetitiveness of chord progressions,and regularity of section durations.
ring to the intra-section characteristics and repetitivenessand novelty referring to the inter-section relationships [1].
More specifically, homogeneity means that musical characteristics (e.g., timbral features such as mel-frequency
cepstrum coefficients (MFCCs)) are consistent within asection. Repetitiveness means that the same sequence ofsome musical elements (e.g., chroma features and chord
progressions) of a section is repeated in sections of thesame class. Novelty means that musical characteristics
change abruptly at a boundary between sections. In addition, regularity of section durations has often been focusedon [8–10] because there are typical lengths such as four oreight measures in popular music.
Most studies on music structure analysis, however, havefocused on only one of the above aspects or consider someof them in a separate and/or ad-hoc manner, as reviewed
in Section 2. Joint computational modeling of these four
aspects is thus the central issue in music structure analysis.
Instead of manually designing segmentation and clustering criteria based on these aspects, we pursue a statistical
approach to data-driven music structure analysis.
In this study, we propose a statistical music structure
analysis method that simultaneously deals with the homogeneity, repetitiveness, and regularity of musical sections in a probabilistic framework (Fig. 1). We formulate a unified probabilistic model called a hierarchical hidden semi-Markov model (HHSMM) that represents the hierarchical generative process of musical sections, chord
sequences, and music audio signals (timbral features andchroma features). This model has two sequences of latentstates in a hierarchical manner. The upper-level sequenceProceedings of the 20th ISMIR Conference, Delft, Netherlands, November 4-8, 2019

2. RELATED WORK
Homogeneity, repetitiveness, novelty, and regularity havebeen considered to be important for music structure analysis. The most standard approach to music structure analysis is to use the self-similarity matrix (SSM) of acoustic
features such as chroma features and MFCCs, whose element represents the acoustic similarity between two timeframes (Fig. 2). In an SSM, the homogeneity, repetitiveness, novelty, and regularity are observed as blockdiagonal structure, short stripes parallel to the diagonalline, grid patterns, and grid intervals, respectively. Oneor some of these four aspects have been used for segmentation and clustering tasks in music structure analysis.
2.1 Segmentation
Foote [2] proposed a novelty-based method that detects
peaks from a time-varying novelty curve obtained by shifting and convoluting a checkerboard kernel along the diagonal elements of an SSM. Jensen [3] attempted to find section boundaries that minimize a homogeneity- and noveltybased cost function. While Goto [19] originally proposed a lag SSM in which repetitions appear not as stripes
but as vertical lines and calculated a novelty curve over

1.00

Regularity

0.75

145.7
0.50

Novelty
116.5
Time (sec)represents a series of section classes following a semiMarkov model with explicit regularity (duration) modelingand the lower-level sequence represents chords following asection-conditioned left-to-right Markov model. To represent the intra-section homogeneity of timbral features, the
MFCCs of a section are assumed to be generated from anupper-level state corresponding to the section. To represent the inter-section repetitiveness of chord progressions,the chroma features of a section are assumed to be sequentially generated from lower-level states. Given a music audio signal as observed data, the whole model can be trained
unsupervisedly using Gibbs sampling and Viterbi training,where unnecessary sections are automatically degenerated
during the Bayesian sparse learning.
The main contribution of this study is to propose a solid
Bayesian approach to music signal analysis based on a
fully generative model that can deal with the homogeneity, repetitiveness, and regularity of sections in a unified
way. This enables unsupervised learning unlike anotherstatistical approach based on deep discriminative models[6–8] that require section annotations for supervised training. Since these two approaches have a mutually complementary relationship, our results open up a door to deep
Bayesian integration of discriminative and generative models in a variational autoencoding framework (audio signal→ sections → audio signal) [22] for further improvement.
Another important contribution is to investigate the statistical characteristics of musical sections estimated by the
proposed method in comparison with representative methods. Our method is shown to be able to yield distributionsof section durations, of the numbers of section classes used
for representing music audio signals, and of the metrical
positions of section boundaries much more similar to thoseof the ground-truth data than the other methods.

0.25

87.4

0.00

Homogeneity
58.3

Repetitiveness

-0.25
-0.50

29.2
-0.75
0.0
0.0

29.2

58.3

87.4
Time (sec)

116.5

145.7

-1.00

Figure 2. Self-similarity matrix (SSM) of MFCC features(part of RWC-MDB-P-2001 No. 25).
time lags, Serrà [4] proposed another novelty curve overtime frames. Peeters and Bisot [5] successfully integrated
these two methods [4, 19] for better segmentation. Ullrich et al. [6] pioneered a supervised approach based on a
convolutional neural network (CNN), which was extended
to deal with both coarse- and fine-level boundary annotations [7]. Sargent et al. [9] pointed out the effectiveness of focusing on the regularity to favor structural segments of comparable size. Maezawa [8] developed a longshort-term memory (LSTM) network with homogeneity-,
repetitiveness-, novelty-, and regularity-based cost functions. In this study we take an unsupervised approachbased on a homogeneity-, repetitiveness-, and regularitybased generative model. To keep the model simple, incorporation the aspect of novelty is left for future work.
2.2 Clustering
Cooper et al. [12] sequentially performed music segmentation [2] and section clustering based on intra- and intersection statistical characteristics. Goodwin et al. [13] attempted to efficiently detect off-diagonal stripes in an SSM
as repetitions using dynamic programming. To deal withrepetitiveness and homogeneity, Grohganz et al. [14] converted a repetitiveness-dominant SSM with off-diagonal
stripes into a homogeneity-dominant SSM with blockdiagonal structure. Nieto et al. [15] used a convex variantof non-negative matrix factorization for section segmentation and clustering. McFee et al. [16] proposed a methodthat encodes repetitive structures into a graph and performs
spectral clustering for graph partitioning.
Several studies took a statistical approach based on
generative models for joint segmentation and clustering.
Aucouturier et al. [11] used a standard hidden Markov
model (HMM). Ren et al. [17] proposed a nonparametric
Bayesian extension of an HMM that can estimate an appropriate number of sections. Barrington et al. [18] proposeda nonparametric Bayesian extension of a switching linear
dynamical system (LDS) that also has the ability of automatic model complexity control. While these methods
mainly focused on the homogeneity, our method simultaneously considers the homogeneity, repetitiveness, andregularity and can incorporate prior knowledge about thestatistical characteristics of sections (e.g., durations andmetrical positions) in a data-driven manner.
269

Proceedings of the 20th ISMIR Conference, Delft, Netherlands, November 4-8, 2019

3. PROPOSED METHOD

Latent States

This section describes the proposed statistical method formusic structure analysis.

duration !
upper "
states

3.1 Problem Specification

lower
states #Observation

Our problem is formulated as follows:
Input: A chroma feature sequence Xc = xc1:B ∈ RB×12
B×12
and an MFCC sequence Xm = xm
obtained
1:B ∈ R
from a given music audio signal. They are computed inunits of beats estimated by a beat tracking method [23].
Output: Boundaries and classes of sections.
Here, B is the number of beats (quarter notes) and a subscript a:b represents the sequence ( a , . . . , b ).
We use 12-dimensional chroma features obtained by aggregating all spectral information of each pitch class intoa single element. We also use 12-dimensional MFCCs astimbral features.
3.2 Model Formulation
The proposed model consists of two-level hierarchicalMarkov chains and an acoustic model as shown in Fig. 3.
The upper-level Markov chain describes the section-levelstructure (i.e., section classes and durations) and the lowerlevel Markov chain describes the internal structure of eachsection. The states of these Markov chains are latent variables that represent abstract musical structure. The acoustic model connects the abstract structure with the observed
musical features (chroma vectors and MFCCs).
3.2.1 Upper-Level Markov Chain
The upper-level Markov chain is an ergodic semi-Markov
model. The sequence of section classes Z = z1:T (zτ ∈{1, . . . , NZ }) and their durations D = d1:T (dτ ∈{1, . . . , ND }) are generated by the model, where T is thenumber of sections, NZ is the number of distinct section
classes, and ND is the maximum duration of a section. The
generative process is described as follows:p(z1 , d1 ) = ρz1 ψd1 ,(1)chromaMFCC

Figure 3. The proposed generative model.
The left-to-right Markov model meets a condition thatthe initial state has kτ,1 = 1 and kτ,t1 ≤ kτ,t2 for t1 < t2 .
We introduce a hyperparameter σ that describes the maximum number of states that may be skipped in a transition; atransition from state k to state k + σ is possible but a larger
skip is forbidden. In this way, the model can describe repetitions with some variations, which is another important
aspect of musical structure. K denotes K1:T .
3.2.3 Acoustic Model
The acoustic model describes the generative processes of12
the chroma features xcb ∈ R12 and MFCCs xm
byb ∈ Rusing output probabilities that are defined conditionally on
the section classes Z and their internal states K. The repetitiveness is represented by applying the same set of output
probabilities to all sections of the same class. The output
probabilities of chroma features χcz,k depend on both Z and
K to represent the sequential structure of chord progressions. To capture the homogeneity of timbre characteristics of each section, the output probabilities of MFCCs χmz
are assumed to depend only on Z. Thus we haveccmmp(xcb , xmb ) = χzb ,kb (xb )χzb (xb ),(4)

where zb and kb are the section class and the internal state
at beat b, respectively. The output probabilities are described as multivariate Gaussian distributions:χcz,k (xcb ) = N (xcb |µcz,k , (Λcz,k )−1 ),(5)where ρz and πzz0 are the initial and transition probabilities
of section classes and ψd is the duration probability.

m
m m
m −1
χm
),
(6)
z (xb ) = N (xb |µz , (Λz )ccwhere µz,k and Λz,k are the mean and precision for themchroma features, and µmz and Λz are defined similarly.

3.2.2 Lower-Level Markov Chain

3.2.4 Prior Distributions

The lower-level Markov chain is a left-to-right Markov
model with NK states. The state sequence of this model
describes the internal structure of a section corresponding
to the chord progression, where each state is expected tocorrespond to a chord. We consider such a Markov chain
for each section class. The model continues state transitions for each beat from the start time of a section until its
duration passes. The state sequence Kτ = kτ,1:dτ (kτ,t ∈{1, . . . , NK }) is generated as follows:

To find an effective number of distinct section classes, weformulate a Bayesian HHSMM by putting conjugate priordistributions. We put Dirichlet prior distributions for the
categorical distributions as follows:p(zτ , dτ |zτ −1 , dτ −1 ) = πzτ −1 zτ ψdτ ,(zτ )p(kτ,t |zτ , kτ,t−1 ) = φkτ,t−1
kτ,t ,(2)(3)

where zτ and dτ are the corresponding section class and(z )duration, and φkkτ0 is the transition probability from state0
k to state k .
270ρ ∼ Dirichlet(aρ ),

(7)

ψ

(8)

π

πz ∼ Dirichlet(a ),

(9)

(z)
φk ∼ Dirichlet(aφ ),(10)

ψ ∼ Dirichlet(a ),(z)where ρ = ρ1:NZ , ψ = ψ1:ND , πz = πz(1:NZ ) , φk =(z)
φk(1:NK ) , and aρ , aψ , aπ , and aφ are Dirichlet parame-

ters. When these parameters are small, the transition probabilities of section classes become sparse. The model can

Proceedings of the 20th ISMIR Conference, Delft, Netherlands, November 4-8, 2019thus capture repetitions regardless of small acoustic variations and remove unnecessary section classes.
Since section durations tend to be the integer multiples
of four measures in popular music (see Fig. 4), such a statistical tendency can be incorporated in the prior distribution. Specifically, we use as aψ the empirical distributionof section durations aψ
emp multiplied by a constant factor.
Since the structure of section classes is quite different overindividual musical pieces, we put uniform Dirichlet priordistributions for their transition probabilities.
Finally, we put Gaussian-Wishart prior distributions on
multivariate normal distributions as follows:

In the forward filtering step, we calculate the probabilitiesζτ,kτ,t recursively as follows:ζτ,kτ,1 = p(kτ,1 , xc1 , xm
1 |zτ , dτ )m= δkτ,1 1 χczτ ,1 (xc1 )χmzτ (x1 ),=

Letting Θ = {ρ, ψ, π, φ, µ, Λ}, we aim to calculate
the posterior distribution p(Z, D, K, Θ|Xc , Xm ). Since
this is analytically intractable, we use the Gibbs sampling
method. We first sample the latent variables Z, D, andK from the distribution p(Z, D, K|Θ, Xc , Xm ) and wethen sample the model parameters Θ from the distributionp(Θ|Z, D, K, Xc , Xm ). Iterating this process, we obtainsamples from the true posterior distribution.

αb (zb , db = b) = ρzb ψdb ωzb (xc1:b , xm
1:b ),p(kτ,dτ |zτ , dτ , xc1:dτ , xm
1:dτ ) ∝ ζτ,kτ,dτ ,(18)
3.3.2 Sampling Model Parameters
We use the Gibbs sampling method for updating the model
parameters as follows:ρ ∼ Dirichlet(aρ + bρ ),p(zB , dB |Xc , Xm ) ∝ αB (zB , dB ).

(13)

When variables zb and db are already sampled, the variables zb0 and db0 at beat b0 = b − db are sampled according
to the probability

πz

ψ

ψ

(19)

),

(20)

ψ ∼ Dirichlet(a + b ),

(21)

(z)
(z)
φk ∼ Dirichlet(aφ + bφk ),cc
Λcz,k ∼ W(Wz,k, νz,k),ccccµz,k |Λz,k ∼ N (mz,k , (βz,k Λcz,k )−1 ),mm
Λmz ∼ W(Wz , νz ),mmm m −1µm),
z |Λz ∼ N (mz , (βz Λz )

(22)
(23)
(24)
(25)
(26)
(z)
φk

where bρ ∈ RNZ , bπz ∈ RNZ , bψ ∈ RND , and b∈
RNK are vectors counting the sampled data. bρz is 1 if z =z
z1 and 0 otherwise, bπ
z 0 counts the number of transitions
0 ψ
from state z to state z , bd counts the number of times that(z)φsampled sections have a duration of d, and bk0k counts thenumber of transitions from state k to state k 0 in the lowerc
level model of section class z. The parameters mcz,k , βz,k,ccWz,k , and νz,k are calculated as follows:ccβz,k= β0c + Nz,k , νz,k
= ν0c + Nz,k ,1
mcz,k = c (β0c mc0 + Nz,k xcz,k ),
βz,k
c
(Wz,k
)−1 = (W0c )−1 + Nz,k Scz,kβ c Nz,k(xc − mc0 )(xcz,k − mc0 )T ,+ c0
β0 + Nz,k z,kwhere we have definedB
X
Nz,k =
δzb z δkb k ,

(27)
(28)

(29)

(30)

b=1
B

p(zb0 , db0 |zb:B , db:B , Xc , Xm ) ∝ αb0 (zb0 , db0 )πzb0 zb .
(14)
Next, the latent variables K are sampled using the sampled Z and D. Each set of variables Kτ is sampled byforward filtering-backward sampling for the lower-level
model of section class zτ . Here we use a beat index t ∈{1, . . . , dτ } considered in relative to the section boundary.

ππz ∼ Dirichlet(a + b

αb (zb , db )
(12)
X
0 0
c
m
=
αb−db (z , d )πz0 zb ψdb ωzb (xb−db +1:b , xb−db +1:b ).
In the backward sampling step, the latent variables Z
and D are sequentially sampled in the reverse order:

(17)

(zτ )
p(kτ,t |zτ , dτ , kτ,t+1:dτ , xc1:dτ , xm
1:dτ ) ∝ ζτ,kτ,t φkτ,t kτ,t+1 .

(11)

z 0 ,d0τm χcz ,k (xct )χmζτ,kτ,t−1 φkτ,t−1zτ (xt ).
kτ,t
τ
τ,t

In the backward sampling step, the latent variables K are
sequentially sampled in the reverse order as follows:

3.3.1 Sampling Latent Variables
We use the forward filtering-backward sampling algorithm
for sampling the upper- and lower-level latent variables Z,
D, and K. We introduce variables zb and db that denote theclass and duration of a section starting at beat b−db +1 andending at beat b. We also define the marginalized output
probabilities for this section ωzb (xcb−db +1:b , xm
b−db +1:b ),which can be calculated by the forward algorithm for the
lower-level Markov chain.
In the forward filtering step for the upper-level model,
we initialize and update the forward variables αb (zb , db ) =p(zb , db , xc1:b , xm
1:b ) as follows:

(z )

X

kτ,t−1m
m
m
m m −1m
mµm) W(Λmz , Λz ∼ N (µz |m0 , (β0 Λz )z |W0 , ν0 ),3.3 Bayesian Learning

(16)





µcz,k , Λcz,k ∼ N (µcz,k |mc0 , (β0c Λcz,k )−1 ) W(Λcz,k |W0c , ν0c ),mmm
where mc0 , β0c , W0c , ν0c , mm
0 , β0 , W0 , and ν0 are hyperparameters.

(15)

ζτ,kτ,t = p(kτ,t , xc1:t , xm
1:t |zτ , dτ )xcz,k =1 X
δzb z δkb k xcb ,
Nz,k

(31)

b=1

B

Scz,k =1 X
δzb z δkb k (xcb − xcz,k )(xcb − xcz,k )T . (32)
Nz,k
b=1

m
m
m
The parameters mm
z , βz , Wz , and νz can be calculated
similarly.

271

Proceedings of the 20th ISMIR Conference, Delft, Netherlands, November 4-8, 2019

3.3.3 Refinements
We introduce two refinements to facilitate the learning process. First, since the samples from the Gibbs sampler are
not necessarily local optimums of the posterior distribution, we apply Viterbi training in the last step of the parameter estimation. Specifically, we apply the Viterbi algorithm (instead of the forward filtering-backward samplingalgorithm) to estimate the latent variables and update themodel parameters to the expectation values of the posterior
probabilities (instead of samples from those probabilities).
It is known that Viterbi training is generally efficient forfinding an approximate local minimum [24].
Second, we introduce a weighting factor wdur (≥ 1) for
the duration probability to enhance its effect. Specifically,
we replace the probability factor ψdb in the forward algorithm (11) and (12) with (ψdb )wdur . Similar replacements
are applied to the Viterbi training step as well as to the final estimation step of latent states explained in Section 3.4.
Increasing the weighting factor has the effect of lowering
the temperature and putting more focus on more frequent
section durations.
3.4 Estimation of Musical Sections
After training the model parameters, we compute the maximum a posteriori (MAP) estimate of the latent variables(musical sections). Specifically, we maximize the posterior
probability p(Z, D|Θ, Xc , Xm ) with respect to latent variables Z and D. This can be solved by integrating out thelower-level states K and applying the Viterbi algorithm for
hidden semi-Markov models [25] to the upper-level model.
4. EVALUATION
4.1 Experimental Conditions
We used the RWC popular music database [26] with structure annotations [27] for evaluation. Out of the 100 pieces
in the data, we used 85 musical pieces in consistent 4/4
time for simplicity. For running the proposed method, we
obtained chroma features using the deep feature extractor [28] and MFCCs using the librosa library [29]. Beat
information was obtained using the madmom library [23].
The empirical distribution aψemp of section durations wastrained using the piece-wise cross validation among the 85
pieces. For parameter estimation, we iterated the Gibbs
sampling 15 times and the Viterbi training 3 times, which
took roughly around 5 times the duration of an input signal
with a standard CPU.
The hyperparameters of the proposed models were set
as follows: NZ = 12, ND = 40, NK = 16, σ = 1,
wdur = 4, aρ = 0.1 · I, aπ = I, aψ = 50 · aψemp ,aφ = I, mc0 = E[Xc ], β0c = 8, W0c = (ν0c cov[Xc ])−1mmmwith ν0c = 96, mm
0 = E[X ], β0 = 4, and W0 =mm −1m
(ν0 cov[X ]) with ν0 = 80, where I denotes a vector
with all entries equal to 1. The first three parameters NZ ,ND , and NK were determined by consulting the statisticsof the annotated data (see Fig. 4). According to the data,
most songs have 12 or less sections and most sections havea length of 40 beats or less. If we expect a section length of
32 beats (8 measures) and a chord duration of 2 beats, the272

MethodVMO [30]
CNMF [15]SCluster [16]Proposed

F0.5 (%)
Fpair (%)
(segmentation) (clustering)
8.72
17.4
23.4
33.0

28.5
41.7
45.5
54.3

Table 1. Evaluation results.
expected number of chords in each section is 16. The valueof σ is set to 1 for simplicity. The other parameters were
determined by a coarse optimization w.r.t. the two evaluation measures explained below. Each parameter was optimized by a grid search, fixing the other parameters. Further
optimization of the parameters is left for future work.
For comparison, we ran variable Markov oracle (VMO)[30], convex non-negative matrix factorization (CNMF)[15], and spectral clustering (SCluster) [16], whichwere available in the music structure analysis framework(MSAF) [31]. We used the default settings in the MSAF.
We evaluated the quality of segmentation and clusteringin the same way as MIREX [32]. The quality of segmentation is evaluated by the F-measure F0.5 of section boundaries [33] defined as follows. An estimated boundary is accepted as correct if there is a boundary in the ground truth
data within the range of ±0.5 seconds. The precision rateis the percentage of correct estimates. The recall rate is thepercentage of true boundaries that are correctly estimated.
The F-measure F0.5 is defined as the harmonic mean of the
precision and recall.
The quality of clustering is evaluated by the pairwise Fmeasure Fpair [34] defined as follows. We compare pairsof frames (with a length of 100 ms) that are labeled with thesame class in an estimation result with those in the groundtruth. The precision, recall, and F-measure are defined as|PE ∩ PA ||PE ∩ PA |Ppair =
, Rpair =
,
(33)
|PE |
|PA |
2Ppair Rpair
Fpair =,
(34)
Ppair + Rpair
where PE denotes the set of similarly labeled frame pairs
in the estimation and PA denotes that in the ground truth.
These values are calculated by the mir_eval library [35].
4.2 Experimental Results
The results in Table 1 show that SCluster had the best F0.5
and Fpair among the three conventional methods. The Fmeasures obtained by VMO were very low and the estimated results included many unnatural short segments(see Fig. 4). This was presumably caused by the implementation in the MASF. In both F0.5 and Fpair , the proposed method significantly outperformed the three compared methods.
Next, let us examine the estimated results more closely(Fig. 4). The distribution of section durations for the proposed model was similar to that of the ground truth. In particular, both distributions have peaks at the 32 beats (eightmeasures) and 16 beats (four measures). In contrast, thedistributions for the results of the other three methods wereProceedings of the 20th ISMIR Conference, Delft, Netherlands, November 4-8, 2019
Ground truth
VMO

CNMF
SCluster
Proposed method
Lower-level states
Reconstructed chroma
Reconstructed MFCCs
Input chroma
Input MFCCs
Time

sec

Figure 4. The left panels show the distributions of section durations, those of metrical positions of section boundaries,
and those of the numbers of section classes in the estimated results and ground truth data. The right figure shows example
results by the proposed and the three existing methods (RWC-MDB-P-2001 No. 29). The lower-level states are obtainedby the Viterbi algorithm and the reconstructed features indicate the mean values of the corresponding output probabilities.
significantly different from that of the ground truth. This
result clearly demonstrates the effect of explicitly modelling the section durations to capture their regularity. We
also found that the distribution of metrical positions of section boundaries for the ground truth data was similar to thatfor the proposed method, but significantly different fromthose for the conventional methods.
The numbers of section classes in the ground truth
data were roughly distributed in the range from eight totwelve. The distribution for the proposed model had a similar shape, though it is slightly shifted to the lower side.
This result demonstrates the nontrivial ability of the proposed method to automatically find the appropriate numberof section classes, even though it often finds the number
smaller than the actual value. On the other hand, the distributions for the other methods were much more sparse; theyfound more or less the same number of section classes for
all the tested pieces. In particular, CNMF and SCluster
estimated too few section classes.
From these analyses, we find that the results of music
structure analysis by the proposed method have much moresimilarity with the human annotated results than the compared existing methods. It is also important to point out
that these results could not be made clear only by looking
at the F-measures. The F-measures are not sufficient forevaluating results of music structure analysis.
We can observe these tendencies in the example results(Fig. 4). Particularly, CNMF and SCluster estimated too
few section classes and irregular section durations. For the
proposed method, we see that sections of the same class
had similar lower-level sequences of latent states. Thissuggests that the model successfully captured repeatedchord progressions in the sections of the same class. We
can also observe that the proposed model often used only apart of lower-level states, which might be improved by im-

posing more constraints on the lower-level Markov chain.
For a fair comparison, we remark that parameters of thethree existing methods were not optimized using our training data. Since we used limited data containing only J-pop
pieces, adapting the parameters of these methods to thisparticular musical style may improve their performance tosome extent. In addition, using the state-of-the-art beattracker [23] to obtain reliable beat information and usingthat as input to those three methods may also improve theiraccuracy. However, it is unlikely that these methods can be
refined to reproduce the aforementioned statistics of sections simply by re-training the parameters.
5. CONCLUSION
We have presented a statistical method for music structure analysis based on a Bayesian HHSMM that describes
intra- and inter-section structures in a unified way. Three ofthe most important aspects of musical sections, homogeneity, repetitiveness, and regularity are incorporated into themodel. Music segmentation and section clustering are performed jointly by unsupervised Bayesian learning of themodel, and musically important characteristics such as therepetitive structure and the distribution of section durations
are incorporated by the Bayesian extension. The experimental results showed that the proposed method achieved
segmentation and clustering accuracies significantly better
than the representative existing methods.
For future work, we plan to refine the model to incorporate the aspect of novelty and to deal with more hierarchies [16] because music has a hierarchical structure, frommotive and phrase to section and section group [36]. Our
unsupervised learning approach is complementary to another approach based on deep discriminative models [6–8].
A promising direction is to integrate these models into avariational autoencoding framework [22].
273Proceedings of the 20th ISMIR Conference, Delft, Netherlands, November 4-8, 2019

6. ACKNOWLEDGEMENTS
This work is supported in part by JST ACCEL No. JPMJAC1602, JSPS KAKENHI Nos. 16H01744, 19K20340,
and 19H04137, and the Kyoto University Foundation.
7. REFERENCES[1] J. Paulus, M. Müller, and A. Klapuri. State of the art
report: Audio-based music structure analysis. In International Society for Music Information Retrieval Conference (ISMIR), pages 625–636, 2010.
[2] J. Foote. Automatic audio segmentation using a measure of audio novelty. In IEEE International Conference on Multimedia and Expo (ICME), pages 452–455,
2000.

[11] J.-J. Aucouturier and M. Sandler. Segmentation ofmusical signals using hidden Markov models. In Audio Engineering Society (AES) Convention, pages 1–8,
2001.
[12] M. Cooper and J. Foote. Summarizing popular music
via structural similarity analysis. In IEEE Workshop on
Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 127–130, 2003.
[13] M. M. Goodwin and J. Laroche. A dynamic programming approach to audio segmentation andspeech/music discrimination. In IEEE International
Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages 309–312, 2004.

[3] K. Jensen. Multiple scale music segmentation using
rhythm, timbre, and harmony. EURASIP Journal on
Applied Signal Processing, 2007(1):159–159, 2007.

[14] H. Grohganz, M. Clausen, N. Jiang, and M. Müller.
Converting path structures into block structures using eigenvalue decompositions of self-similarity matrices. In International Society for Music Information
Retrieval Conference (ISMIR), pages 209–214, 2013.

[4] J. Serrà, M. Müller, P. Grosche, and J. Arcos. Unsupervised detection of music boundaries by time series structure features. In International Society for Music Information Retrieval Conference (ISMIR), pages
1613–1619, 2012.

[15] O. Nieto and T. Jehan. Convex non-negative matrix
factorization for automatic music structure identification. In IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), pages 236–
240, 2013.

[5] G. Peeters and V. Bisot. Improving music structure
segmentation using lag-priors. In International Society
for Music Information Retrieval Conference (ISMIR),
pages 337–342, 2014.

[16] B. McFee and D. P. W. Ellis. Analyzing song structure with spectral clustering. In International Society
for Music Information Retrieval Conference (ISMIR),
pages 405–410, 2014.

[6] K. Ullrich, J. Schlüter, and T. Grill. Boundary detection
in music structure analysis using convolutional neural networks. In International Society for Music Information Retrieval Conference (ISMIR), pages 417–422,
2014.

[17] L. Ren, D. Dunson, S. Lindroth, and L. Carin. Dynamic
nonparametric Bayesian models for analysis of music.
Journal of the American Statistical Association (JASA),
105(490):458–472, 2008.

[7] T. Grill and J. Schlüter. Music boundary detection using neural networks on combined features and twolevel annotations. In International Society for Music
Information Retrieval Conference (ISMIR), pages 531–
537, 2015.
[8] A. Maezawa. Music boundary detection based on a
hybrid deep model of novelty, homogeneity, repetition and duration. In IEEE International Conferenceon Acoustics, Speech and Signal Processing (ICASSP),
pages 206–210, 2019.
[9] G. Sargent, F. Bimbot, and E. Vincent. Estimating thestructural segmentation of popular music pieces under regularity constraints. IEEE Transactions on Audio,
Speech and Language Processing (TASLP), 25(2):344–
358, 2017.
[10] F. Kaiser and G. Peeters. A simple fusion method ofstate and sequence segmentation for music structure
discovery. In International Society for Music Information Retrieval Conference (ISMIR), pages 257–262,
2013.
274

[18] L. Barrington, A. B. Chan, and G. Lanckriet. Modeling music as a dynamic texture. IEEE Transactions
on Audio, Speech and Language Processing (TASLP),
18(3):602–612, 2010.
[19] M. Goto. A chorus section detection method for musical audio signals and its application to a music listening
station. IEEE Transactions on Audio, Speech, and Language Processing (TASLP), 14(5):1783–1794, 2006.
[20] J. Paulus and A. Klapuri. Music structure analysis
using a probabilistic fitness measure and a greedy
search algorithm. IEEE Transactions on Audio, Speechand Language Processing (TASLP), 17(6):1159–1170,
2009.
[21] T. Cheng, J. B. L. Smith, and M. Goto. Music structure boundary detection and labelling by a deconvolution of path-enhanced self-similarity matrix. In International Society for Music Information Retrieval Conference (ISMIR), pages 106–110, 2018.
[22] D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In International Conference on Learning
Representations (ICLR), pages 1–14, 2014.

Proceedings of the 20th ISMIR Conference, Delft, Netherlands, November 4-8, 2019[23] S. Böck, F. Korzeniowski, J. Schlüter, F. Krebs, and
G. Widmer. madmom: A new python audio and musicsignal processing library. In ACM International Conference on Multimedia (ACMMM), pages 1174–1178,
2016.

[35] C. Raffel, B. McFee, E. J. Humphrey, J. Salamon,
O. Nieto, D. Liang, and D. P. W. Ellis. mir_eval: Atransparent implementation of common MIR metrics.
In International Society for Music Information Retrieval Conference (ISMIR), 2014.

[24] A. Allahverdyan and A. Galstyan. Comparative analysis of Viterbi training and maximum likelihood estimation for HMMs. In Advances in Neural Information
Processing Systems (NIPS), pages 1674–1682, 2011.

[36] F. Lerdahl and R. Jackendoff. A Generative Theory ofTonal Music. MIT Press, 1983.

[25] S.-Z. Yu. Hidden semi-Markov models. Artificial Intelligence, 174(2):215–243, 2010.
[26] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka.
RWC music database: Popular, classical and jazz music databases. In International Conference on Music Information Retrieval (ISMIR), pages 287–288, 2002.
[27] M. Goto. AIST annotation for the RWC music
database. In International Conference on Music Information Retrieval (ISMIR), pages 359–360, 2006.
[28] Y. Wu and W. Li. Automatic audio chord recognition with MIDI-trained deep feature and BLSTM-CRF
sequence decoding model. IEEE/ACM Transactions
on Audio, Speech and Language Processing (TASLP),
27(2):355–366, 2019.
[29] B. McFee, C. Raffel, D. Liang, D. P. W. Ellis,
M. McVicar, E. Battenberg, and O. Nieto. librosa: Audio and music signal analysis in python. In Python in
Science Conference, pages 18–24, 2015.
[30] C.-I. Wang and G. J. Mysore. Structural segmentation
with the variable Markov oracle and boundary adjustment. In IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), pages 291–
295, 2016.
[31] O. Nieto and J. P. Bello. Systematic exploration ofcomputational music structure research. In International Society for Music Information Retrieval Conference (ISMIR), 2016.
[32] A. F. Ehmann, M. Bay, J. S. Downie, I. Fujinaga,
and D. De Roure. Music structure segmentation algorithm evaluation: Expanding on MIREX 2010 analyses
and datasets. In International Society for Music Information Retrieval Conference (ISMIR), pages 561–566,
2011.
[33] D. Turnbull, G. Lanckriet, E. Pampalk, and M. Goto. A
supervised approach for detecting boundaries in music
using difference features and boosting. In International
Conference on Music Information Retrieval (ISMIR),
pages 51–54, 2007.
[34] M. Levy and M. Sandler. Structural segmentation ofmusical audio by constrained clustering. IEEE Transactions on Audio, Speech, and Language Processing(TASLP), 16(2):318–326, 2008.
275