deepGTTM-I&II: Local Boundary and Metrical
Structure Analyzer based on Deep Learning
Technique
Masatoshi Hamanaka1 , Keiji Hirata2 , and Satoshi Tojo3
1

Kyoto University
masatosh@kuhp.kyoto-u.ac.jp
2
Future University Hakodate
hirata@fun.ac.jp
3
JAIST
tojo@jaist.ac.jp

Abstract. This paper describes an analyzer for detecting local grouping boundaries and generating metrical structures of music pieces basedon a generative theory of tonal music (GTTM). Although systems forautomatically detecting local grouping boundaries and generating metrical structures, such as the full automatic time-span tree analyzer, havebeen proposed, musicologists have to correct the boundaries or strong
beat positions due to numerous errors. In light of this, we use a deeplearning technique for detecting local boundaries and generating metrical structures of music pieces based on a GTTM. Because we only have
300 pieces of music with the local grouping boundaries and metricalstructures analyzed by musicologist, directly learning the relationshipbetween the scores and metrical structures is diﬃcult due to the lack oftraining data. To solve this problem, we propose a multi-task learning
analyzer called deepGTM-I&II based on the above deep learning technique to learn the relationship between scores and metrical structures in
the following three steps. First, we conduct unsupervised pre-training ofa network using 15,000 pieces of music in a non-labeled dataset. After
pre-training, the network involves supervised ﬁne-tuning by back propagation from output to input layers using a half-labeled dataset, which
consists of 15,000 pieces of music labeled with an automatic analyzer
that we previously constructed. Finally, the network involves supervised
ﬁne-tuning using a labeled dataset. The experimental results indicatethat deepGTTM-I&II outperformed previous analyzers for a GTTM interms of the F-measure for generating metrical structures.
Keywords: generative theory of tonal music (GTTM), local groupingboundary, grouping structure, metrical structure, deep learning.

1

Introduction

We propose an analyzer for automatically detecting local grouping boundaries
and generating metrical structures of music pieces based on a generative theory

of tonal music (GTTM) [1]. A GTTM is composed of four modules, each of whichassigns a separate structural description to a listener’s understanding of a pieceof music. These four modules output a grouping structure, metrical structure,
time-span tree, and prolongational tree. Since the detection of the local groupingboundaries and generation of metrical structures is the early stage in a GTTM,
an extremely accurate analyzer will make it possible to improve the performanceof all later analyzers.
We previously constructed several analyzers, such as the automatic timespan tree analyzer (ATTA) [2] and fully automatic time-span tree analyzer(FATTA) [3], that enable us to detect local grouping boundaries. However, theperformance of these analyzers is inadequate in that musicologists have to correct
the boundaries due to numerous errors.
Our deepGTTM-I&II is based on a deep learning technique [4] to improve
the performance of detecting local grouping boundaries and generating metricalstructures of music pieces. Unsupervised training in the deep learning of deeplayered networks called pre-training helps in supervised training, which is called
ﬁne-tuning [5].
Our goal was to develop a GTTM analyzer that enables the output of theresults obtained from analysis that were the same as those obtained by musicologists based on deep learning by learning the analysis results obtained by themusicologists. We had to consider the following three issues in constructing thisanalyzer.
Multi-task learning
A model or network in a simple learning task estimates the label from an
input feature vector. However, local grouping boundaries can be found inmany note transitions. Also, a strong beat can be found in many beat positions. Therefore, we consider a single learning task for estimating whether
one note transition can be a boundary (a strong beat).
A problem in detecting local grouping boundaries or strong beats can thenbe solved using multi-task learning.
Subsection 4.3 explains multi-task learning by using deep learning.
Large-scale training data
Large-scale training data are needed to train a deep-layered network, andlabels are not needed in pre-training the network. Therefore, we collected
15,000 pieces of music formatted in musicXML from Web pages that wereintroduced in the MusicXML page of MakeMusic Inc citeMakeMusic. We
needed labeled data to ﬁne-tune the network. Although we had 300 pieces
with labels in the GTTM database [7], this number was too small to enable
the network to learn.
Subsection 4.1 explains how we collected the data and how we got the network to learn eﬀectively with a small dataset.
GTTM rules
A GTTM consists of multiple rules, and a note transition that is applied tomany rules tends to be a local grouping boundary in the analysis of localgrouping boundaries. Similarly, a beat that is applied to many rules tends

to be a strong beat in the analysis of the metrical structure. As a resultof analysis by musicologists, 300 pieces of music in the GTTM databasewere not only labeled with local grouping boundaries and metrical structures
but also labeled with applied positions of preference rules (PRs). Therefore,
the applied positions of PRs were helpful clues in detecting local groupingboundaries and strong beats.
Subsection 4.3 explains how the network learned with the PRs.
Sequential vs. recurrent models
There are two types of models, i.e., recurrent and sequential, that can beused for GTTM analysis. The recurrent neural network provides recurrent
models, which are suitable for analyzing a metrical structure in which cyclical change results in strong and weak beats. However, the recurrent neuralnetwork is diﬃcult to train, and training time is very long. On the other
hand, sequential models, such as deep belief networks (DBNs), are not suitable for detecting the repetition of strong beats. However, the deepGTTM-I
DBN of our analyzer is very simple and performs well in detecting the localgrouping boundary of a GTTM.
Therefore, we chose two DBNs for GTTM analysis. Subsection 4.2 explains
how these DBNs are trained for analyzing metrical structures.
Hierarchical metrical structure
A hierarchical metrical structure is generated by iterating the choice of thenext-level structure. The next level structure is recursively generated using
the previous structure. However, when we use learning with a single standard
network of deep learning, it is diﬃcult to lean a higher-level structure because
many network representations are used for learning a lower-level structure.
Subsection 4.2 explains how to learn a higher level structure.
The results obtained from an experiment indicate that our GTTMI&II involving multi-task learning using the deep learning technique outperformed previous GTTM analyzers.
The paper is organized as follows. Section 2 describes related work and Section 3 explains our analyzer called deepGTTM-I&II. Section 4 explains how weevaluated the performance of deepGTTM-I&II, and Section 5 concludes with asummary and overview of future work.

2

Related Work

We now brieﬂy look back through the history of cognitive music theory. The imprecation realization model (IRM) proposed by Narmour abstracts and expresses
music according to symbol sequences from information from a score [8, 9]. Recently, the IRM has been implemented on computers, and its chain structures
can be obtained from a score [10]. The Schenkerian analysis analyzes deeper
structures called “Urline” and “Ursatz” from the music surface [11]. Short segments of music can be analyzed through Schenkerian analysis on a computer [12].
There is another approach that constructs a music theory for adopting computer
implementation [13, 14].

We consider a GTTM to be the most promising of the many theories thathave been proposed [8, 9, 11, 13, 15] in terms of its ability to formalize musicalknowledge because a GTTM captures the aspects of musical phenomena basedon the Gestalt occurring in music and is presented with relatively rigid rules.
The main advantage of analysis by using a GTTM is that it can acquire tree
structures called time-span and prolongation trees.
We have been constructing both analyzers and an application of a GTTM
for more than a decade (Fig. 1) citeHamanaka 2016. The horizontal axis in Fig.
1 indicates years. The analyzers we developed are above the timeline.
2.1

Analyzers for GTTM based on full parameterization

We ﬁrst constructed a grouping structure analyzer and metrical structure analyzer (Figs. 1a and b). We developed the ATTA (Fig. 1c) [2] by integrating agrouping structure analyzer and a metrical analyzer. We extended the GTTM
by full externalization and parameterization and proposed a machine-executable
extension of the GTTM, exGTTM. We implemented exGTTM on a computerthat we call the ATTA . The ATTA has 46 adjusted parameters to control thestrength of each rule. The ATTA we developed enables us to control the priorityof rules, which enables us to obtain extremely accurate groupings and metrical structures. However, we need musical knowledge that musicologists have toproperly tune the parameters.
The FATTA [3] (Fig. 1d) does not have to tune the parameters because it
automatically calculates the stability of structures and optimizes the parameters
so that the structures would be stable. It achieved excellent analysis results for
metrical structures, but the results for grouping structures and time-span trees
were unacceptable.
We constructed an interactive GTTM analyzer [22](Fig. 1e) that enables
seamless changes in the automatic analysis and manual editing processes because
it was diﬃcult to construct an analyzer that could output analysis results in thesame way as musicologists. The interactive GTTM analyzer is still used to collect
GTTM analysis data, and anyone can download and use it for free [23].
However, all these analyzers [2, 3, 22, 23] have problems. Musical knowledge isrequired for the ATTA to tune the parameters, and the FATTA performs poorly.
2.2

Analyzers for GTTM based on statistical learning

The σGTTM analyzer [24] (Fig. 1f) enables us to automatically detect localgrouping boundaries by using a decision tree. Although σGTTM performed better than the FATTA, it was worse than the ATTA after the ATTA parameters
had been tuned.
The σGTTMII analyzer [25] (Fig. 1g) involves clustering steps for learning
the decision tree and outperforms the ATTA if we can manually select the best
decision tree. Although σGTTMII performed the best in detecting groupingboundaries, it was diﬃcult to select the proper decision tree without musicalknowledge.

Fig. 1. Related work on analysis and application systems for GTTM
TA

ϭ

ϭ

Ϭ

B2nb−1
Ϭ

ϭ

B3nd
Ϭ

Ϭ

B3nc−1B2nbϭdŚĞŬŝŶĚŽĨ'WZ
n
3a



dĂƌŐĞƚƚƚƌŝďƵƚĞď/ĨĂůŽĐĂůŐƌŽƵƉŝŶŐďŽƵŶĚĂƌǇŝƐďĞƚǁĞĞŶŶŽƚĞƐ͕ďсϭ͕ŽƚŚĞƌǁŝƐĞď сϬ͘B2na+1 , B2nb+1 , B3na+1 , B3nb+1 , B3nc+1 , B3nd+1

B2na−1 , B2nb−1 , B3na−1 , B3nb−1 , B3nc−1 , B3nd−1 ,

n
2b

ϭ

Ϭ

B ࠉ
, B , B , B3nb , B3nc , B3nd ,n
2aB2na+1ϭdŚĞƉŽƐŝƚŝŽŶďĞƚǁĞĞŶŶŽƚĞƐ

b=0ŽŶĚŝƚŝŽŶĂůƚƚƌŝďƵƚĞ

B2na

B3nc−1



b=0

Ϭ

Ϭ

ϭ

ϭ

Ϭ

B3nc

9LR OLQ ,

%DVVR

9LROD

9LRO LQ ,,&Allegro## c wf##c œ œ œ œ œ œ œ œfB ## c œ œ œ œ œ œ œ œ
f? ## c œ œ œ œ œ œ œ œ
f

&

‰
‰

Œ
Œ

œ œ

œ

œ

œ œ œ œ œ œ œ œœ œ œ œ œ œ œ œ œ œ œ œ œ œ œ œ

œ œ œ

œ œ

œ

œ œ œ œ œ œ œ œ w

œ œ œ œ.

œœ œœ œœ œœ œœœ œœ œœ œ œrœ ˙œ œ œ œ œ œ œ œ

TA TB

TC
TD

䠄㻟䠅㻯㼛㼙㼎㼕㼚㼕㼚㼓 㼠㼣㼛 㼙㼑㼘㼛㼐㼕㼑㼟

LB

TC
LA
䠄㻞䠅㻹㼑㼘㼛㼐㼥㼐㼕㼢㼕㼟㼕㼛㼚 㼞㼑㼐㼡㼏㼠㼕㼛㼚

TB

TD

Melody D

Melody BTD

TC 12㻔㼏㻕 㻹㼑㼘㼛㼐㼥 㼐㼕㼢㼕㼟㼕㼛㼚㼍㼘㻭㼡㼓㼙㼑㼚㼠㼍㼠㼕㼛㼚

TA

TC5

㻹㼑㼘㼛㼐㼥 㼐㼕㼢㼕㼟㼕㼛㼚㼍㼘 㼞㼑㼐㼡㼏㼠㼕㼛㼚㻹㼑㼘㼛㼐㼥 㼐㼕㼢㼕㼟㼕㼛㼚㼍㼘 㼍㼡㼓㼙㼑㼚㼠㼍㼠㼕㼛㼚

Input melody A

Melody C5

œ.
œ
œ

œ

œœœœ œœœ œ œœ œœœœœ œ œjœœ œ œ œ œ œ œ œ œ œœ œ œ œ œ œ œjœ œwœ œ œ œ œ œ œ œ œ œœ œ œ œ œ œ œ

j

œ

œ

œ œ œ œ œ œ œ œœ œ œ œ œ œ œ œœ œ œ œ œ œ œ œœ ˙DƵƐŝĐyD> ďǇŵƵƐŝĐŽůŽŐŝƐƚƐ
;ŚŽŵŽƉŚŽŶǇͿr#œ ˙œ œ œ œ œ œ œrr
œ œ œ œrr
œ œ œ œdŝŵĞͲƐƉĂŶ
yD>;WŽůǇƉŚŽŶǇͿœœœ œ œ œ œ œ nœRnœ œ œ œ œ œ œœ œ œœœœ œ œœ œ œœœœ œ œrr#œ n œ œ œ .

r
œ n œ œr œ .

dŝŵĞͲƐƉĂŶ ƚƌĞĞĂŶĂůǇǌĞƌœ œ œ œ œ œ œ œ

œ

œ œ
œ œ

r
#œ ˙

‰ œ

œ
œ

œ œ œ œœ œ œ œœ œJœ œ œ œœ œ œ œ

œ œ #œ œ œdŝŵĞͲƐƉĂŶyD>;ŚŽŵŽƉŚŽŶǇͿ͘͘͘DĞƚƌŝĐĂůyD>͘DĞƚƌŝĐĂůƐƚƌƵĐƚƵƌĞĂŶĂůǇǌĞƌ

DĂŶƵĂůĂŶĂůǇƐŝƐ͘͘͘͘'ƌŽƵƉŝŶŐƐƚƌƵĐƚƵƌĞĂŶĂůǇǌĞƌœ œ œ œ œ œ œ œ œœ œœ œœ œœ œœœ œœ œœ œ œ

r
œ ˙

͘͘
͘͘

,ĂƌŵŽŶŝĐyD>,ĂƌŵŽŶŝĐĂŶĂůǇǌĞƌ

DĂŶƵĂůĂƌƌĂŶŐĞŵĞŶƚďǇŵƵƐŝĐŽůŽŐŝƐƚƐ

'ƌŽƵƉŝŶŐyD>㼏㼛㼙㼙㼛㼚

TC1 2 TD 6

TC5

TA TB

TD 6

TD 6

TD

TB

Extrapolative melody

Interpolative melody㼕㼚㼒㼛㼞㼙㼍㼠㼕㼛㼚 㼛㼒 㼠㼔㼑 㼙㼑㼘㼛㼐㼕㼑㼟㻔㼎㻕 㻹㼑㼘㼛㼐㼥 㼐㼕㼢㼕㼟㼕㼛㼚㼍㼘㼞㼑㼐㼡㼏㼠㼕㼛㼚TC㻔㼍㻕㻸㼕㼚㼗㼕㼚㼓㻔㼐㻕 㻯㼛㼙㼎㼕㼚㼕㼚㼓 㼠㼣㼛 㼙㼑㼘㼛㼐㼕㼑㼟

TC5

TA TB




S 6KDNH*XLWDU +'

K ̴*770,,,
EDVHGRQ3&)*>@L S*770 EDVHGRQ3&)*>@&RPSRVLQJV\VWHPR 6KDNH*XLWDU

Input melody B

Melody D6

TD6



3RO\SKRQLF0XVLF7LPH6SDQ7UHHϬDƵƐŝĐyD>
Wdd;ŵŽŶŽƉŚŽŶǇͿDƵƐŝĐyD>;WŽůǇƉŚŽŶǇͿ WĂƌƚĚŝǀŝĚĞƌ



B3nd

J ̴*770⋜ >@3$77$3RO\SKRQLF 0XVLF
7LPH6SDQ7UHH$QDO\]HUQ 0HORG\PRUSKLQJ>@ P 0HORG\0HORG\H[WUDSRODWLRQ([SHFWDWLRQ >@



Yes

TSRPR5䛾㐺⏝䠄㻝䠅㻸㼕㼚㼗㼕㼚㼓 㼠㼔㼑 㼏㼛㼙㼙㼛㼚 㼕㼚㼒㼛㼞㼙㼍㼠㼕㼛㼚 㼛㼒 㼙㼑㼘㼛㼐㼕㼑㼟

Melody E

Melody A

Melody CTCO 5HSURGXFLQJPXVLF>@



No

䝦䝑䝗䛜2䛴௨ୖ

ḟ
ḟ䛾㝵
䛾
䛾㝵
㝵ᒙ
ᒙ

䝍䜲䝮䝇䝟䞁ᮌ䞉ᢿ⠇ᵓ㐀䛾ᏳᐃᗘDTSRPR5 + DGPR7
2

D GPR7GPR7䛾㐺⏝DTSRPR5b =1
ŶĚEŽĚĞB

͙

N 5HQGHULQJ>@M 6XPPU\]DWLRQ >@

Ditime-span

⌧ᅾ
⌧
⌧ᅾ
ᅾ䛾ᵓ
䛾
䛾ᵓ
ᵓ㐀
㐀

(䝦䝑䝗䛾ᙉ䛥)Yes

TSRPR1,3,4,8,9䛾㐺⏝

ᢿⅬ䛜2䛴௨ୖ
No

Time-spanXML

ḟ
ḟ䛾㝵
䛾㝵
ᒙ䛾䝦
䝑䝗
䛾
㝵ᒙ䛾
ᒙ
䛾
䛾䝦
䝦䝑䝗
䝑
䝗
䛾
䛾㑅ᢥ
㑅
㑅ᢥ
ᢥ

No

MPR1,2,3,4,5䛾㐺⏝ [time]

MPR10䛾㐺⏝䚸ᵓ㐀䛾㑅ᢥ

m=4
m=5

m 1
ḟ
ḟ䛾㝵
䛾
䛾㝵
㝵ᒙ
ᒙ m=2
䛾
䛾ೃ⿵
ೃ
ೃ⿵
⿵
m=3

=

Dilow-level(ᢿⅬ䛾ᙉ䛥)Yes

GPR1, 2, 3, 4, 5, 6 䛾㐺⏝䜾䝹䞊䝥୰䛻ᒁᡤⓗቃ⏺䛜䛒䜛

⌧
⌧ᅾ䛾
ᅾ
ᅾ䛾
䛾ᵓ㐀
ᵓ
ᵓ㐀
㐀

MetricalXML

䝦䝑
䝦
䝦䝑䝗
䝑
䝑䝗䛾
䝗
䝗䛾
䛾ᙉ䛥
ᙉ
ᙉ䛥
䛥䛾⟬
䛾
䛾⟬
⟬ฟ
ฟ

䝍
䝍䜲䝮
䝍䜲
䜲
䜲䝮䝇
䝮
䝮䝇䝟
䝇
䝇䝟䞁
䝟
䝟䞁ᮌ
䞁
䞁ᮌ
ᮌ
ศᯒ
ศ
ศᯒჾ
ᯒ
ᯒჾ
ჾ

ḟ
ḟ䛾㝵
䛾
䛾㝵
㝵ᒙ䛾
ᒙ
ᒙ䛾
䛾
ᢿ
ᢿ⠇ᵓ
⠇
⠇ᵓ
ᵓ㐀䛾
㐀
㐀䛾㑅
䛾
䛾㑅
㑅ᢥ
ᢥ

ᒁᡤ
ᒁ
ᒁᡤⓗ
ᡤ
ᡤⓗ
ⓗ䛺ᢿ
䛺
䛺ᢿⅬ
ᢿ
ᢿⅬ䛾
Ⅼ
Ⅼ䛾
䛾ᙉ䛥
ᙉ
ᙉ䛥
䛥
䛾⟬
䛾
䛾⟬ฟ
⟬
⟬ฟ
ฟ

ᢿ
ᢿ⠇
ᵓ㐀
ᯒჾ
ᢿ⠇ᵓ
⠇
⠇ᵓ
ᵓ㐀ศ
㐀
㐀ศᯒ
ศ
ศᯒ
ᯒჾ
ჾ

GroupingXML

㧗
㧗ḟ䛾
ḟ
ḟ䛾
䛾ቃ⏺
ቃ
ቃ⏺䛾
⏺
⏺䛾
䛾᳨ฟ
᳨
᳨ฟ
ฟ

䝖䝑
䝖䝑䝥
䛻䜘䜛
ศ๭
䝖
䝑
䝑䝥䝎
䝥
䝥䝎䜴
䝎
䝎䜴
䜴䞁
䞁䛻䜘
䛻
䜘
䜘䜛
䜛ศ๭
ศ
๭

[time]

ϭ^ƉůŝƚƚŝŶŐdĞƐƚ

͙



䝍䜲䝮䝇䝟䞁ᮌศᯒჾ䛾ㄪ⠇ྍ⬟䛺
䝟䝷䝯䞊䝍

ᒁᡤ
ᒁ
ᒁᡤ
ᡤⓗቃ
ⓗ
ⓗቃ
ቃ⏺
⏺

ATTA: 䝍䜲
䞁ᮌศ
ჾ
䝍
䝍䜲䝮
䜲
䜲䝮䝇
䝮
䝮䝇䝟
䝇
䝇䝟
䝟䞁ᮌ
䞁
ᮌ
ᮌศᯒ
ศ
ศᯒ
ᯒჾ

͙

᭱㐺໬

Bi

Ϭ

͙

ᢿ⠇ᵓ㐀ศᯒჾ
䛾ㄪ⠇ྍ⬟䛺
䝟䝷䝯䞊䝍MusicXML
䜾
䜾䝹
䜾䝹䞊
䝹
䝹䞊䝢
䞊䝢䞁
ᵓ㐀
㐀
䞊
䝢
䝢䞁䜾
䞁
䞁䜾ᵓ
䜾
䜾ᵓ
ᵓ㐀
ศ
ศᯒ
ศᯒჾ
ᯒ
ᯒჾ
ჾ
ᒁ
ᒁᡤⓗ
ᡤ
ᡤⓗ
ⓗቃ⏺
ቃ
ቃ⏺䛾
⏺
⏺䛾
䛾᳨ฟ
᳨
᳨ฟ
ฟ (ቃ⏺䛾῝䛥) GPR1, 2, 3, 6 䛾㐺⏝

B2nb

͙

᭱㐺໬

䜾䝹䞊䝢䞁䜾ᵓ㐀ศᯒჾ䛾ㄪ⠇ྍ⬟䛺
䝟䝷䝯䞊䝍

᭱㐺໬

䝟
䝷䝯䞊
໬㒊
䝟䝷䝯
䝷
䝯
䝯䞊䝍
䞊
䞊䝍᭱
䝍
䝍᭱㐺
᭱
᭱㐺
㐺໬㒊
໬
㒊

FATTA: ᏶
඲⮬ື
䝍䜲䝮
ᯒჾ
᏶඲⮬
඲
⮬
⮬ື
ື䝍䜲
䝍
䜲
䜲䝮䝇
䝮
䝮䝇
䝇䝟䞁
䝟
䝟䞁ᮌ
䞁
䞁ᮌศ
ᮌ
ᮌศ
ศᯒჾ
ᯒ
ჾ

ϭ

n
2a

I ̴*770 >@͙
͙D *URXSLQJVWUXFWXUHDQDO\]HUE 0HWULFDOVWUXFWXUHDQDO\]HUG )$77$)XOO$XWRPDWLF7LPHVSDQ
7UHH$QDO\]HU>@F $77$$XWRPDWLF 7LPHVSDQ7UHH$QDO\]HU>@H ,QWHUDFWLYH*770$QDO\]HU>@͙The σGTTMIII analyzer [26] (Fig. 1h) enables us to automatically analyze
time-span trees by learning with a time-span tree of 300 pieces of music fromthe GTTM database [7] based on probabilistic context-free grammar (PCFG).
The σGTTMIII analyzer performed the best in acquiring time-span trees. The
pGTTM anlayzer [27] (Fig. 1i) also uses PCFG, and we used it to attempt
unsupervised learning. The main advantages of σGTTMIII and pGTTM arethat they can learn the contexts in diﬀerence hierarchies of the structures (e.g.,beats were important in the leaves of time-span trees, or chords were importantnear the roots of the trees.).
However, all these analyzers [24–27] have problems with regard to detecting local grouping boundaries. The σGTTM III and pGTTM analyzers are focused on acquiring time-span trees and cannot acquire local grouping boundaries.
Musical knowledge is required for σGTTM II to select the decision tree. Since
σGTTM and σGTTM II use rules that musicologists applied, they cannot work
as standalone analyzers. For example, information on parallel phrases is needed
when detecting local grouping boundaries because parallel phrases create parallel structures in a GTTM. However, σGTTM and σGTTM II do not haveprocesses for acquiring parallel phrases.
In light of this, we introduce a deep learning analyzer for detecting the localgrouping boundaries and generating hierarchical metrical structures of musicpieces based on a GTTM.
2.3

Application systems by using analysis results of GTTM

We constructed applications systems, which are given under the time-line in
Fig. 1, to use the results from GTTM analysis. A time-span or prolongation
tree provides a summarization of a piece of music, which can be used as therepresentation of an abstraction, resulting in a music retrieval system [17] (Fig.
1j). It can also be used for performance rendering [18] (Fig. 1k) and reproducing music [19](Fig. 1l). The time-span tree can also be used for melody prediction [20](Fig. 1m) and melody morphing [21](Fig. 1n). Figures 1o and p illustrate
a demonstration system for the melody morphing method that changes the morphing level of each half bar by using the values from the accelerometer in the
iPad/iPhone/iPod Touch. When the user stops moving the iPhone/iPod Touch,the unit plays the backing melody of ”The Other Day, I Met a Bear (The Bear
Song)”. When the user shakes it vigorously, it plays heavy soloing. When theuser shakes it slowly, it plays a morphed melody between the backing and heavysoloing.
These systems currently need a time-span tree analyzed by musicologists
because our analyzers do not perform optimally.
2.4

Melody segmentation

Because conventional approches of melody segmentation, such as the Grouperof the Melisma Music Analyzer by Temperley [28] and the local boundary detection model (LBDM) by Cambouropoulos [29], require the user to make manualadjustments to the parameters, they are not completely automatic. Although
Temperley [30] has also used a probabilistic model, it has not been applied to
melody segmentation. The unsupervised learning model (IDyOM) proposed by
Pearce et al. makes no use of the rules of music theory with regard to melodic
phrases, and it has performed as well as Grouper and LBDM [31]. However, as
deepGTTM-I&II statistically and collectively learn all the rules for the grouping
structure analysis of a GTTM, we expect that deepGTTM-I&II will performbetter than an analyzer that only uses statistical learning.
2.5

Beat tracking

The metrical structure analysis in a GTTM is a kind of beat tracking. Current
methods based on beat tracking [32–35] can only acquire the hierarchical metricalstructure in a measure because they do not take into account larger metricalstructures such as two and four measures.

3

GTTM and Its Implementation Problems

Figure 2 shows local grouping boundaries, a grouping structure, metrical structure, timespan tree, and prolongational tree.
The grouping structure is intended to formalize the intuitive belief that tonal
music is organized into groups that are in turn composed of subgroups. These
groups are presented graphically as several levels of arcs below a music staﬀ.
The metrical structure describes the rhythmical hierarchy of a piece of music by identifying the position of strong beats at the levels of a quarter note,
half note, measure, two measures, four measures, and so on. Strong beats areillustrated as several levels of dots below the music staﬀ.
3.1

Preference rules

There are two types of rules in a GTTM, i.e., ”well-formedness” and ”preference”. Well-formedness rules (WFRs) are necessary for the assignment of astructure and restrictions on the structure. When more than one structure satisﬁes the WFRs, the PRs indicate the superiority of one structure over another.
There are seven grouping PRs (GPRs) (GPRs): GPR1 (alternative form),
GPR2 (proximity), GPR3 (change), GPR4 (intensiﬁcation), GPR5 (symmetry), GPR6 (parallelism), and GPR7 (time-span and prolongational stability).
The GPR2 has two cases: (a) (slur/rest) and (b) (attack-point). The GPR3
has four cases: (a) (register), (b) (dynamics), (c) (articulation), and (d) (length).
There are ten metrical PRs (MPRs) (MPRs): MPR1 (parallelism), MPR2
(strong beat early), MPR3 (event), MPR4 (stress), MPR5 (length), MPR6
(bass), MPR7 (cadence), MPR8 (suspension), MPR9(time-span interaction),and MPR10 (binary regularity). The MPR5 has six cases: (a) pitch-event, (b)dynamics, (c) slur, (d) articulation, (e) repeated pitches, and (f) harmony.

Prolongaon tree

Time-span tree

Metrical structureGrouping structure

Fig. 2. Grouping structure, metrical structure, time-span tree, and prolongation tree

3.2

Conflict between rules

Because there is no strict order for applying PRs, a conﬂict between rules often
occurs when applying PRs, which results in ambiguities in analysis.
Figure 3(a) outlines a simple example of the conﬂict between GPR2b (attackpoint) and GPR3a (register). GPR2b states that a relatively greater intervalof time between attack points initiates a grouping boundary. The GPR3a statesthat a relatively greater diﬀerence in pitch between smaller neighboring intervals initiates a grouping boundary. Because GPR1 (alternative form) has strong
preference for note 3 alone not forming a group, a boundary cannot be perceived
at both 2-3 and 3-4.
Figure 3(b) shows an example of the conﬂict between MPRs 5c and 5a. The
MPR5c states that a relatively long slur results in a strong beat, and MPR5astates that a relatively long pitch-event results in a strong beat. Because metrical
WFR 3(MWFR3) states that strong beats are spaced either two or three beatsapart, a strong beat cannot be perceived at both onsets of the ﬁrst and second
notes.
We expect to learn the rule application and priority of rules by inputting awhole song with labels of the applied rules to a deep-layered network.
3.3

Ambiguous rule definition

Some rules in a GTTM are expressed with ambiguous terms. The GPR4 andMPR5 are deﬁned as follows as examples.
The GPR4 (Intensification), where the eﬀects selected by GPRs 2 and3 are relatively more pronounced, a larger-level group boundary may beplaced.

(a) Conflict between GPRs(b) Conflict between MPRs

3a 2b
5c 5a

Candidate 1
Candidate 2

Candidate 1

Violate GPR1

Candidate 2

Fig. 3. Examples of conﬂict between PRs

The MPR5 (Length) has preference for a metrical structure in whicha relatively strong beat occurs at the inception of either
a. a relatively long pitch-event,b. relatively long duration of a dynamic,c. relatively long slur,d. relatively long pattern of articulation,e. relatively long duration of a pitch in the relevant levels of the timespan reduction, or
f. relatively long duration of a harmony in the relevant levels of thetime-span reduction (harmonic rhythm).
The term ”relatively” in this sense is ambiguous. The sentence also contains
the phrase ”more pronounced”, but the comparison is unclear. Another example
is that a GTTM has rules for selecting proper structures when discovering similar
melodies (called parallelism) but does not deﬁne similarity. The GPR6 andMPR1 are deﬁned as follows as examples.
The GPR6 (parallelism), when two or more segments of the music canbe construed as parallel, they preferably form parallel parts of groups.
The MPR1 (parallelism), when two or more groups or parts of groups
can be construed as parallel, they preferably receive a parallel metricalstructure.

3.4

Context dependency

To solve the problems discussed in Subsections 3.2 and 3.3, we proposed themachine executable extension of GTTM (exGTTM) and ATTA [2]. Figure 4
gives an example of an application of MPR4, 5a, 5b, and 5c in the exGTTM
and ATTA. By conﬁguring the threshold parameters T j (j = 4, 5a, 5b, and 5c),
we can control whether each rule is applicable. However, proper values of theparameter depend on the piece of music and level of hierarchy in the metricalstructure. Therefore, automatic estimation of proper values of the parameters is
diﬃcult.

…
^^5b
5b
5c
velo
2µvelo

^
4
5a
5b

^
4
5a
5b

^^
5b5b
5c

^
4
5a
5b

^
4
5a
5b

^^
5b5b
5c

Current structure

T4
[i]

valu
2µvalu

T5a
[i]

vol
2µvol

T5b
[i]

slur
2µslurT5c[i]Fig. 4. Application of MPR4, 5a, 5b, and 5c in ATTA

3.5

Less precise explanation of feedback link

A GTTM has various feedback links from higher-level structures to lower-levelones. For example, MPR9 is deﬁned as follows.
The MPR9 (Time-span interaction) has preference for a metricalanalysis that minimizes conﬂict in the time-span reduction.
However, no detailed description and only a few examples are given. Other feedback links in the GTTM rules are not explicit. For example, analyzing the resultsof a time-span tree strongly aﬀects the interpretation of chord progression, andvarious rules are related to chord progression, e.g., MPR7 (Cadence) requires
a metrical structure in which cadences are metrically stable.
For complete implementation of a GTTM based on deep learning, we haveto introduce the feedback link by using the recurrent neural network; however,
we do not focus on the feedback link in this paper.

4

deepGTTM-I&II: Local Grouping Boundary andMetrical Strucutre Analyzer based on Deep Learning

We introduced deep learning into deepGTTM-I&II to analyze the structure of a
GTTM and solve the problems described in Subsections 3.2, 3.3 and 3.4. There
are two main advantages of introducing deep learning.
Learning rule applications
We constructed a deep-layered network that can output whether each ruleis applicable on each note transition by learning the relationship between
the scores and positions of applied grouping PRs with the deep learningtechnique.
Previous analysis systems based on GTTM were constructed by a humanresearcher or programmer. As described in Subsection 3.3, some rules in a
GTTM are very ambiguous and the implementations of these rules might
diﬀer depending on the person.
However, deepGTTM-I&II is a learning based analyzer, where its quality
depends on the training data and trained network.
Learning priority of rules
The FATTA only determine the priority of rules from the stability of thestructure. The σGTTM and σGTTMII analyzers only determine the priorityof rules from applied rules. They do not work well because the priority ofrules depends on the context of a piece of music.
The input of the network in deepGTTM-I&II, on the other hand, is the score
and the network learns the priority of the rules as the weight and bias of the
network based on the context of the score.
This section describes how we analyzed the local grouping boundaries andmetrical structure by using deep learning.
4.1

Datasets for training

Three types of datasets were used to train the network, i.e., a non-labeled dataset
for pre-training, half-labeled dataset, and labeled dataset for ﬁne-tuning (Fig. 5).

tĞďĂƵƚŽƉŝůŽƚ

ϭϱ͕ϬϬϬƉŝĞĐĞƐ;ŵƵƐŝĐyD>Ϳdd

Ͳ DƵƐŝĐyD>䠄ĂͿEŽŶͲůĂďĞůĞĚĚĂƚĂƐĞƚ

'ddDĚĂƚĂďĂƐĞͲ DƵƐŝĐyD>Ͳ ƉƉůŝĞĚƌƵůĞƐ

䠄ďͿ,ĂůĨͲůĂďĞůĞĚĚĂƚĂƐĞƚͲ DƵƐŝĐyD>Ͳ ƉƉůŝĞĚƌƵůĞƐ
Ͳ >ŽĐĂůŐƌŽƵƉŝŶŐďŽƵŶĚĂƌŝĞƐ

䠄ĐͿ>ĂďĞůĞĚĚĂƚĂƐĞƚ

ϯϬϬƉŝĞĐĞƐ
ǆϭϮŬĞǇƐ
ǆϳŬŝŶĚŽĨŶŽƚĞǀĂůƵĞƐ

Fig. 5. Non-labeled, half-labeled, and labeled datasets

Non-labeled dataset
The network in pre-training learned the features of the music. A large-scale
dataset with no labels was needed. Therefore, we collected 15,000 pieces ofmusic formatted in musicXML from Web pages that were introduced on
the musicXML page of MakeMusic Inc. [6] (Fig. 5a). The musicXMLs were
downloaded in the following three steps.

a. Web autopilot script made a list of urls that probably downloaded musicXMLs in ﬁve links from the musicXML page of MakeMusic Inc.
b. The ﬁles in the url list were downloaded after they had been omitted
because they were clearly not musicXML.
c. All the downloaded ﬁles were opened using the script, and ﬁles that were
not musicXML were deleted.
Half-labeled dataset
The network in ﬁne-tuning learned with the labeled dataset. We had 300
pieces of music with a labeled dataset in the GTTM database, which included
musicXML with positions of local grouping boundaries, and positions towhich the GPRs were applied. However, 300 pieces are insuﬃcient for deeplearning.
Consequently, we constructed a half-labeled dataset. We automatically added
the labels of thirteen applied rules of GPR2a, 2b, 3a, 3b, 3c, 3d, andMPR2, 3, 4, 5a, 5b, 5c, and 5d, because these rules could be uniquely
applied as a score. We used the ATTA to add labels to these rules (Fig. 5b).
With the ATTA, the strength of the beat dependent on each MPR can be
expressed as Di j (j = 2, 3, 4, 5a, 5b, 5c, and 5d, 0 ≤ Di j ≤ 1). For example,
MPR4 is deﬁned in a GTTM as follows.
The MPR4 (Event) has preference for a metrical structure in whichbeats of level Li that are stressed are strong beats of Li .
We formalized Di 4 as follows.
{
1 veloi > 2 × µvelo × T 4
Di 4 =(1)0
else,
where veloi is the velocity of a note from beat i, µvelo is the average of veloi ,and T j (0 ≤ T j ≤ 1) are the threshold parameters to control those that
determine whether the rules are applicable (Di j = 1) (Di j = 0). We used 1 as
the threshold parameter value (T j = 1, where j = 2, 3, 4, 5a, 5b, 5c, and 5d).
Labeled dataset
We collected 300 pieces of 8-bar-long, monophonic, classical music and asked
people with expertise in musicology to analyze them manually with faithfulregard to the MPRs. These manually produced results were cross-checkedby three other experts.
We artiﬁcially increased the labeled dataset because 300 pieces of music in
the GTTM database were insuﬃcient for training a deep-layered network.
First, we transposed the pieces for all 12 keys. We then changed the length of
the note values to two times, four times, eight times, half time, quarter time,and eighth time. Thus, the total labeled dataset had 25,200 (= 300x12x7)
pieces of music (Fig. 5c).

4.2

Deep belief networks

We used deep belief networks (DBN) for detecting local grouping boundaries
and generating metrical structures.
Figure 6 outlines the structure of this DBN, which we call deepGTTM-I , fordetecting local grouping boundaries. The inputs of deepGTTM-I are the onset
time, oﬀset time, pitch, and velocity of note sequences from musicXML There
are 11 outputs of deepGTTM-I to enable multi-tasking learning, i.e., ten GPRs
(GPR2a, 2b, 3a, 3b, 3c, 4, 5, 6, and 7) and a local grouping boundary.

䞉䞉䞉

䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉

䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉

䞉䞉䞉䞉

䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉

䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉

䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉

䞉䞉䞉 䞉䞉䞉䞉

䞉䞉䞉
'WZϲ

䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉

ŽƵŶĚĂƌǇ
'WZϮĂ,ŝĚĚĞŶ>ĂǇĞƌn Ͳϭ䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉,ŝĚĚĞŶ>ĂǇĞƌϮ&ƵůůǇĐŽŶŶĞĐƚĞĚ
䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉

䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉

䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉

䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉,ŝĚĚĞŶ>ĂǇĞƌϭ䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉,ŝĚĚĞŶ>ĂǇĞƌnKŶƐĞƚƚŝŵĞKĨĨƐĞƚƚŝŵĞWŝƚĐŚ
sĞůŽĐŝƚǇ

Fig. 6. DBN for detecting local grouping boundaries

Figure 7 outlines the structure of the DBN we call deepGTTM-II to generatea metrical structure . The inputs of deepGTTM-II are the onset time, oﬀset time,
pitch, and velocity, and grouping structure manually analyzed by musicologists.
Each hierarchical level of the grouping structure is separately inputted by a noteneighboring the grouping boundary as 1; otherwise, 0. There are eight outputsof deepGTTM-II to enable multi-tasking learning in each hierarchical level ofthe metrical structure, i.e., seven MPRs (MPR2, 3, 4, 5a, 5b, 5c, and 5d) and
one level of the metrical structure. Individual outputs have two units, e.g., rulesthat were not applicable (=0) and rules that were applicable (=1), or weak beats(=0) and strong beats (=1).

DĞƚƌŝĐĂůƐƚƌƵĐƚƵƌĞůĞǀĞůhDĞƚƌŝĐĂůĚŽƚƐ
DWZϮ
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉

䞉䞉䞉
DWZϱĚ

䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉

,ŝĚĚĞŶůĂǇĞƌn+h䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉&ƵůůǇĐŽŶŶĞĐƚĞĚƚŽŚŝĚĚĞŶůĂǇĞƌn+h-1DĞƚƌŝĐĂůƐƚƌƵĐƚƵƌĞůĞǀĞůϭDĞƚƌŝĐĂůĚŽƚƐ
DWZϮ

䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉

䞉䞉䞉
DWZϱĚ

䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉

,ŝĚĚĞŶ>ĂǇĞƌn+1䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉&ƵůůǇĐŽŶŶĞĐƚĞĚƚŽŚŝĚĚĞŶůĂǇĞƌnDĞƚƌŝĐĂůƐƚƌƵĐƚƵƌĞůĞǀĞůϬDĞƚƌŝĐĂůĚŽƚƐ
DWZϮ
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉
䞉䞉䞉

䞉䞉䞉
DWZϱĚ

䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉
䞉䞉䞉䞉

,ŝĚĚĞŶůĂǇĞƌn Ͳϭ䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉,ŝĚĚĞŶůĂǇĞƌϮ䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉,ŝĚĚĞŶůĂǇĞƌn&ƵůůǇĐŽŶŶĞĐƚĞĚ
䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉,ŝĚĚĞŶůĂǇĞƌϭ

䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉

䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉

䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉
䞉䞉䞉䞉䞉

䞉䞉䞉䞉䞉

䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉䞉KŶƐĞƚƚŝŵĞKĨĨƐĞƚƚŝŵĞWŝƚĐŚ
sĞůŽĐŝƚǇ
'ƌŽƵƉŝŶŐďŽƵŶĚĂƌŝĞƐ
>ĞǀĞůϯ>ĞǀĞůϮ>ĞǀĞůϭ>ĞǀĞůϬ^ĐŽƌĞ'ƌŽƵƉŝŶŐƐƚƌƵĐƚƵƌĞ
>ĞǀĞůϬ>ĞǀĞůϭ>ĞǀĞůϮ>ĞǀĞůϯ

Fig. 7. DBN for generating a metrical structure

A metrical structure consists of hierarchical levels, and we added one hidden
layer to generate the next structure level. We used logistic regression to connect
the ﬁnal hidden layer (n,n+1,..., n+h) and outputs. All outputs shared thehidden layer from 1 to the ﬁnal hidden layer. The network was learned in thefour steps below. The order of music pieces was changed at every epoch in all
steps.
4.3

Multi-dimensional multi-task learning

Our deepGTTM-I&II consists of a very complex network. The ﬁne-tuning of localgrouping boundaries and one level of a metrical structure involves multi-task
learning. The ﬁne-tuning of each PR also involved multi-task learning. Therefore,
the ﬁne-tuning of PRs involves multi-dimensional multi-task learning (Fig. 8).

DƵůƚŝͲƚĂƐŬůĞĂƌŶŝŶŐ

DƵůƚŝͲĚŝŵĞŶƐŝŽŶĂůůĞĂƌŶŝŶŐ

^ĞůĞĐƚĞĚĨƌŽŵƚŽƉƚŽďŽƚƚŽŵ
DWZϮ
Žƌ
Žƌ
Žƌ
Žƌ

DWZϮ
DWZϱď
DWZϰ
DWZϱĚ
DWZϯ
䠖
䠖

KƌĚĞƌŽĨDWZƐǁĂƐƌĂŶĚŽŵůǇƐŚƵĨĨůĞĚ

Žƌ

Fig. 8. Multi-dimensional multi-task learning

Multi-task learning The processing ﬂow for the multi-task learning of a GPR
or local grouping boundaries involves the following four steps.

Step 1: The order of the pieces of training data is randomly shuﬄed and a pieceis selected from top to bottom.
Step 2: The note transition of the selected piece is randomly shuﬄed and anote transition is selected from top to bottom.
Step 3: Back propagation from output to input is carried out based on whether
the note transition had a boundary or the rule was applied (=1) or not (=0).
Step 4: The next note transition or the next piece in steps 1 and 2 is repeated.
The processing ﬂow for the multi-task learning of an MPR or metrical dots
involved four steps.

Step 1: The order of the music pieces of training data is randomly shuﬄed anda piece is selected from top to bottom.

Step 2: The beat position of the selected piece is randomly shuﬄed and a beatposition is selected from top to bottom.
Step 3: Back propagation from output to input is carried out based on whether
the beat position had a strong beat or the rule was applied (=1) or was not(=0).
Step 4: The next beat position or the next piece in steps 2 and 3 is repeated.
Multidimensional multi-task learning The processing ﬂow for the multidimensional multi-task learning of PRs involves the following three steps.
Step 1: The order of PRs is randomly shuﬄed and a rule is selected from top
to bottom.
Step 2: Multi-task learning of the selected PR is carried out.
Step 3: The next rules in step 1 are repeated.

5

Experimental Results

We evaluated the Fmeasure of deepGTTM-I&II by using 100 music pieces fromthe GTTM database; the remaining 200 pieces were used to train the network.
The Fmeasure is given by the weighted harmonic mean of precision P (proportion of selected dots that are correct) and recall R (proportion of correct dotsthat were identiﬁed).
×R
F measure = 2 × P
P +R

(2)

Tables 1 and 2 summarize the results of deepGTTM-I and deepGTTM-II,
respectively, for a network that had 11 layers with 3000 units.
The results in the tables indicate that deepGTTM-I&II outperformed previous analyzers in terms of the F-measure.
The ATTA has adjustable parameters and its performance changes depending on the parameters. For the default parameter, we used the middle value inthe parameter range [2]. The σGTTMII analyzer selected the decision tree andthe performance changes on the selected tree. The performances of the ATTAand σGTTMII changed depending on the parameters or decision trees. Table 1
indicates the best performance was achieved by manual editing. However, the
FATTA, σGTTM, and deepGTTM-I have no parameters for editing.
These results show that deepGTTM-I&II performed extremely robustly.

6

Conclusion

We developed an analyzer for detecting local grouping boundaries and generatinga metrical structure called deepGTTM-I&II that is based on deep learning. The
following three points are the main results of this study.

Table 1. Performances of deepGTTM-I, ATTA, σGTTM

ATTA with manual editing of parameters
σGTTM
σGTTM with manual selection of decision tree
deepGTTM-I

Precision P Recall R F measure
0.74
0.44
0.55
0.47
0.74
0.57
0.68
0.92
0.78
0.78
0.81
0.80

Table 2. Performances of deepGTTM-II, ATTA, and FATTA
deepGTTM-II
Melodies
1. Grande Valse Brillante
2. Moments Musicaux
3. Trukish March
4. Anitras Tanz
5 Valse du Petit Chien
Total (100 melodies)

0.94
1.00
0.98
0.90
0.99
:
0.96

ATTA
ATTA
FATTA
(Default (Conﬁgured
prameters) parameters)
0.88
0.93
0.88
0.95
1.00
1.00
0.91
0.96
0.96
0.82
0.86
0.82
0.87
0.92
0.95
:
:
:
0.84
0.90
0.88

Music analyzer based on deep learning
It has been revealed that deep learning is strong for various tasks. We demonstrated that deep leaning is also strong for music analysis. We will attempt
to implement other music theories based on deep learning. Although wecollected 300 pieces of music and analyzed the results of a GTTM by musicologists, the labeled dataset of these pieces was not suﬃcient for learninga deep-layered network. We therefore used our previous GTTM analyzer
called ATTA to prepare three types of datasets, non-labeled, half-labeled,and labeled, to learn the network.
High-accuracy GTTM analyzer without manual editing
Previous GTTM analyzers, such as the ATTA and σGTTM, require manual
editing; otherwise, the performance will be much worse. The Fmeasure s of
GTTM analyzers without manual editing, such as the FATTA, σGTTM,
σGTTMIII, and pGTTM, are too low (under 0.8). However, deepGTTMI&II exhibited extremely high performance, which indicates the possibilityof practical use in GTTM applications [17–21]. We plan to implement theentire GTTM analysis process based on deep learning.
Multi-dimensional multi-task learning
We constructed a multi-dimensional multi-task learning analyzer that eﬃciently learns the grouping boundaries and hierarchical level of the metricalstructure and PRs by sharing the network. Multi-dimensional multi-task
learning is expected to be applied to other data that have a hierarchy andtime series such as ﬁlm [36] and discussion [37]. After a network that had 11

layers with 3000 units had been learned, the deepGTTM-I&II outperformed
the previously developed analyzers in terms of the Fmeasure .
This work was one step in implementing a GTTM based on deep learning.
The remaining steps are to implement time-span reduction analysis and prologational reduction analysis of a GTTM based on deep learning. The following
two problems remain. One is generating tree structures because time-span andprolongation tree structures are more complex than local grouping boundaries
or a hierarchical metrical structure. The other problem is the lack of training
samples because there are many combinations of tree structures, and an unlearned sample sometimes appears in test data. We will attempt to solve these
problems and make it possible to construct a complete GTTM system based on
deep learning.
At the current stage, we are not able to understand the details on why deeplearning works extremely well for metrical analysis in a GTTM. Thus, we alsoplan to analyze a network after a metrical structure is learned.
Acknowledgments This work was supported by JSPS KAKENHI Grant Number 25700036, 16H01744, 23500145.

References
1. Lerdahl, F., and Jackendoﬀ, R.: A Generative Theory of Tonal Music. MIT Press,
Cambridge (1983)
2. Hamanaka, M., Hirata, K., and Tojo, S.: Implementing ‘A Generative Theory ofTonal Music’, Journal of New Music Research, 35:4, 249–277 (2006)
3. Hamanaka, M., Hirata, K., and Tojo, S.: FATTA: Full Automatic Time-span Tree
Analyzer, In: Proceedings of the 2007 International Computer Music Conference(ICMC2007), pp. 153–156 (2007)
4. Hinton, G. E., Osindero, S., and Teh Y.-W.: A fast learning algorithm for deep
belief nets, Neural Computation,18:7, 1527–1554 (2006)
5. Erhan, D., Bengio, Y., Courville, A., Manzagol, P.-A., Vincent, P., and Bengio,
S.: Why does unsupervised pre-training help deep learning? Journal of Machine
Learning Research, vol. 11, 625–660 (2010)
6. MakeMusic Inc.: Finale, http://www.finalemusic.com/, see at 2017-1-4.
7. Hamanaka, M., Hirata, K., and Tojo, S.: Music Structural Analysis Database basedon GTTM, In: Proceedings of the 2014 International Society for Music Information
Retrieval Conference (ISMIR2014), pp. 325–330 (2014)
8. Narmour E: The Analysis and Cognition of Basic Melodic Structure. University of
ChicagoPress (1990).
9. Narmour E: The Analysis and Cognition of Melodic Complexity. The Universityof ChicagoPress (1992)
10. Yazawa, S. and Hamanaka, M. and Utsuro, T.: Melody Generation System basedon a Theory of Melody Sequences, In: Proceedings of ICAICTA2014, pp. 347–352(2014)
11. Schenker, H. Der frei Satz. Vienna: Universal Edition, 1935. Published in English as
Free Composition, translated and edited by E. Oster, New York: Longman (1979)

12. Marsden, A.: Software for Schenkerian Analysis, In: Proceeding of International
Computer Music Conference (ICMC2011), pp. 673–676 (2011)
13. Temperley, D.: The Congnition of Basic Musical Structures. MIT Press, Cambridge(2004)
14. Lerdahl, F.: Tonal Pitch Space, Oxford University Press (2001)
15. Cooper, G., and Meyer, L. B.: The Rhythmic Structure of Music. The Universityof Chicago Press (1960)
16. Hamanaka ,M., Hirata, K., and Tojo, S.: Implementing Methods for Analysing Music Based on Lerdahl and Jackendoﬀ’s Generative Theory of Tonal Music, Computational Music Analysis, 221–249, Springer (2016)17. Hirata, K., and Matsuda, S.: Interactive Music Summarization based on Generative
Theory of Tonal Music. Journal of New Music Research, 32:2, 165–177 (2003)
18. Hirata, K., and Hiraga, R.: Ha-Hi-Hun plays Chopin’s Etude, Working Notes of
IJCAI-03 Workshop on Methods for Automatic Music Performance and their Applications in a Public Rendering Contest, pp. 72–73 (2003)
19. Hirata, K., and Matsuda, S.: Annotated Music for Retrieval, Reproduction, andSharing, In: Proceeding of International Computer Music Conference (ICMC2004),
pp. 584–587 (2004)
20. Hamanaka, M., Hirata, K., and Tojo, S.: Melody Expectation Method based on
GTTM and TPS, In: Proceeding of the 2008 International Society for Music Information Retrieval Conference (ISMIR2008), pp. 107–112 (2008)
21. Hamanaka, M., Hirata, K., and Tojo, S.: Melody Morphing Method based on
GTTM, In: Proceeding of the 2008 International Computer Music Conference(ICMC2008), pp. 155–158 (2008)
22. Hamanaka, M., Hirata, K., and Tojo, S.: Interactive GTTM Analyzer, In: Proceedings of the 10th International Conference on Music Information Retrieval Conference (ISMIR2009), pp. 291-296 (2009)23. Hamanaka, M.: Interactive GTTM Analyzer/GTTM Database, http://gttm.jp,
seen on 2017-1-4.
24. Miura, Y., Hamanaka, M., Hirata, K., and Tojo, S.: Use of Decision Tree to Detect
GTTM Group Boundaries, In: Proceedings of the 2009 International Computer
Music Conference (ICMC2009), pp. 125–128 (2009)
25. Kanamori, K., and Hamanaka, M.: Method to Detect GTTM Local Grouping
Boundarys based on Clustering and Statistical Learning, In: Proceedings of the2014 International Computer Music Conference (ICMC2014), pp. 125–128 (2014)
26. Hamanaka, M., Hirata, K., and Tojo, S.: σGTTM III: Learning-based Time-span
Tree Generator Based on PCFG, In: Proceedings of the 11th International Symposium on Computer Music Multidisciplinary Research (CMMR 2015), pp. 303-317(2015)
27. Nakamura, E., Hamanaka, M. Hirata, K., and Yoshii, K.: Tree-Structured Probabilistic Model of Monophonic Written Music Based on the Generative Theoryof Tonal Music, In: Proceedings of The 41st IEEE International Conference on
Acoustics, Speech and Signal Processing (ICASSP2016), pp. 276–280 (2016)
28. Temperley, D.: The Melisma Music Analyzer. http://www.link.cs.cmu.edu/
music-analysis/ (2003), seen on 2017-1-4.
29. Cambouropoulos, E. :The Local Boundary Detection Model (LBDM) and its application in the study of expressive timing, In: Proceedings of the International
Computer Music Conference (ICMC2001), pp. 290-293 (2001)
30. Temperley, D.: Music and Probability. Cambridge: The MIT Press (2007)

31. Pearce, M. T., Müllensiefen, D., and Wiggins, G. A.: A comparison of statistical andrule-based models of melodic segmentation, In: Proceedings of the International
Conference on Music Information Retrieval (ISMIR2008), pp. 89-94 (2008)
32. Rosenthal, D.: Emulation of human rhythm perception, Computer Music Journal,
16:1, 64–76 (1992)
33. Goto, M.: An Audio-based Real-time Beat Tracking System for Music With or
Without Drum-sounds, Journal of New Music Research, 30:2, 159–171 (2001)
34. Dixon, S.: Automatic Extraction of Tempo and Beat from Expressive Performance,
Journal of New Music Research, 30:1, 39–58 (2001)
35. Davies, M. and Bock, S.: Evaluating the Evaluation Measures for Beat Tracking,
In: Proceedings of the International Conference on Music Information Retrieval(ISMIR2014), pp. 637-642 (2014)
36. Takeuchi, S. and Hamanaka, M.: Structure of the ﬁlm based on the music theory,
in JSAI2014, 2014, 1K5-OS-07b-4 (in Japanese).
37. Oshima, T., Hamanaka, M., Hirata, K., Tojo, S. and Nagao, K.: Development of
discussion structure editor for discussion mining based on muisc theory, in IPSJSIG DCC, 2013, 7 pages (in Japanese).