A MUSIC STRUCTURE INFERENCE ALGORITHM BASED ON MORPHOLOGICAL ANALYSIS

Music structure refers to the description of the long term organization of a music piece through a sequence of structural segments. A structural segment can be defined by its structural borders (a start time, an end time) and a label reflecting the similarity of its music content compared to the other segments’. Its duration is typically around 16 s and more. This document presents the music structure estimation system submitted to MIREX’s structural segmentation task in 2012. It is composed of three steps : feature extraction, structural border estimation and segment labeling. First, the system produces a sequence of chroma vectors [6] expressed at the snap scale [1] (section 1). This sequence is used to calculate a segmentation criterion based on a morphological model of the structural segments [2] (section 2.1). The structural border estimation is performed by searching the segmentation with lowest cost, which combines this criterion and a regularity constraint (section 2.2). The segments are then labeled by clustering according to their similarity, through the minimization of an adaptive model selection criterion (section 3). 1. FEATURE EXTRACTION The extraction of the sequence of chroma vectors of size 12 used to describe the music content of the piece is performed by means of the “Chroma Toolbox” by Muller and Ewert [6]. We use the CP features regularly and a hop of 0.1 s. Then, they are expressed at the snap scale. The snap is here defined as the multiple of a beat whose period is closer to 1 s. The snap scale is synchronous to the downbeat scale, and they are often equal in practice. The beat and downbeat estimations are performed thanks to the MATLAB implementation by Davies et al. [4, 5]. The downbeat estimator is tuned so as to consider 4 beats per bar. We associate to each snap the mean of the CP features contained in the window centered on the snap that lasts the This document is licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 License. http://creativecommons.org/licenses/by-nc-sa/3.0/ c © 2012 The Authors. duration of the snap period. 2. STRUCTURAL BORDER ESTIMATION 2.1 Morphological model We assume that a structural segment can be characterized by its inner organization, according to its musical layers (timbre, harmony, melody .). In this scope we consider the system and contrast model by Bimbot et al. [2]. It considers that each structural segment aimed is built from an group of typically four morphological elements of four snaps, we note {a1, a2, a3, a4}. The three first elements are related by simple transformations f and g so as a2 = f(a1) and a3 = g(a1). The fourth element can either follow the logic of the three elements and then form a system (a4 = f(g(a1))) or on the contrary contrasts with it (a4 = δ(f(g(a1))) where δ 6= id). Note that we assume that the relevant layers for structural analysis can vary from one structural segment to another. However, in much cases, either f = id or g = id, or both. This leads to observe usual morphological motives like aaaa, abab, aabb in the case of systems with no contrast, or aaab, abac, aabc in the case of systems ending with a contrast. These motives can be extended to the case where the identity function id is replaced by “close to identity” functions id′ (aaa′b, aba′c, aa′bc, .). More information on this model can be found in [3]. 2.2 Segmentation criterion The aim is to evaluate for each time unit considered the likelihood that it corresponds to the beginning of a system. We assume that at least one of the relations (f or/and g) between the elements of a system equals the identity function. For each snap t ∈ [1, T ] of a music piece, we consider the analysis window of size N = 16 snaps so as to consider three morphological elements starting from t (a1, a2, a3), and one morphological element before this snap (a0) as represented in figure 1. We consider that the size of each morphological element is Nm = 4 snaps. The criterion Φ we consider in this work results from the linear combination of two quantities : Φ = λ1σSystem + λ2σContrast (1) Figure 1. Analysis window used for segmentation criterion calculation, containing the sequence of features x. It is composed of 4 small windows, each one related to the position of a morphological element ai of size 4 snaps, i = {0, 1, 2, 3}. with λ1 and λ2 ∈ R learnt on a training database 1 . σSystem(t) quantifies the likelihood that t corresponds to the start time of a system through the analysis of the similarity of the morphological elements a2 and a3 with regard to a1. Be x = {Xn}1≤n≤N the sequence of features contained in the analysis window related to snap t. Let us define X = {X0, X1, X2, X3}, with vector Xi = {x1+iNm , . , xNm+iNm} for i = {0, 1, 2, 3}, and Y = [Y1, Y2] = [(X2 −X1), (X3 −X1)]. Xi contains the sequence of features of ai, and Yj is the squared distance between the features of aj+1 and a1, for each of its dimensions. We have : σSystem(t) = ∑Tm j=1 min(Y1(j), Y2(j)) ||X1|| (2) where ||Xi|| corresponds to the l2 norm of vector Xi. σSystem(t) allows to evaluate the contribution of X1 to explain either X2 or X3 according to the various dimensions of the features. A high contribution implies that a system is likely to begin at snap t in the piece. In the scope of the system and contrast model, a structural segment is likely to begin at t if a2 and/or a3 is similar to a1. This is considered through σSystem(t). If the preceding structural segment is different from the current one, the third and the fourth morphological elements differ from a1. If the two segments are the same, then the third element may be similar to a1, but the fourth one generally differs from it. We therefore introduce σContrast(t) which evaluates the dissimilarity between the morphological element preceding snap t we note a0, and a1. We choose to formulate it as follows : σContrast(t) = cotan(X0, X1) (3) where cotan(Xi, Xj) corresponds to the cotangent of the angle between vectors Xi and Xj . 2.3 Regularity constraint We consider a regularity constraint in the structural border estimation to favor segmentations with segments close to a 1 RWC Popular database with structural annotations from [1] were used for parameter tuning. typical segment size or structural pulse τ = 16 snaps . Let m be the size of a structural segment : Ψα(m) = | m τ − 1| (4) with α ∈ R a factor which controls the convexity of this function. The use of a non-convex function will favor segmentations with a majority of segments of size equal to τ and few segments whose size is far from this structural pulse. On the opposite, a convex function loosen this constraint. 2.4 Performing the structural border estimation The segmentation criterion and the regularity constraint are combined through a linear combination to form a segmentation cost C : C = (1− λ3)Φ + Ψα (5) with λ3 ∈ [0, 1]. We use the Viterbi algorithm described in [8] to find the segmentation with the lowest segmentation cost.