GTTM DATABASE AND MANUAL TIME-SPAN
TREE GENERATION TOOL
Masatoshi Hamanaka
RIKEN
masatoshi.hamanaka@riken.jp

Keiji Hirata
Future University Hakodate
hirata@fun.ac.jp

Satoshi Tojo
JAIST
tojo@jaist.ac.jp

ABSTRACTThis paper describes the publication of a musical structure analysis database and a tool for manually generatingtime-span trees on the basis of the generative theory of
tonal music (GTTM). We previously analyzed 300 pieces
of music with the analysis database and the analysis editor on the basis of the GTTM. However, the length ofeach piece was about eight bars (which is short), and theconventional analysis editor did not work for pieces ofmusic with a large number of bars and notes, which thus
took a lot of time to manually edit. Therefore, we developed a tool for manually generating time-span tree analyses for the GTTM that can manipulate a piece of musicwith a large number of bars and notes. Four musicologists
developed an analysis database of 50 musical pieces of
32 bars in length by using the manual generation tool.
The experimental results show that the average editingtime with the time-span tree generation tool is shorter
than that with the previous tool.

1. INTRODUCTION
For over 15 years, we have been implementing music
analyzers on the basis of the generative theory of tonal
music (GTTM), which was proposed by Lerdahl andJackendoff [1–5]. The GTTM is composed of four modules, each of which assigns a separate structural description to a listener’s understanding of a piece of music. The
GTTM outputs a grouping structure, a metrical structure,a time-span tree, and a prolongational tree (Fig. 1).
The grouping structure is intended to formalize theintuitive belief that tonal music is organized into groups
that com-prise subgroups. The metrical structure presents
the rhythmical hierarchy of the musical piece by identifying the position of strong beats at the levels of quarter
notes, half notes, one measure, two measures, fourmeasures, and so on. The time-span tree is a binary tree
and has a hierarchical structure presenting the relative
structural importance of notes that differentiate the essential parts of the melody from the ornamentation. The prolongational tree is a binary tree that expresses the structure of tension and relaxation in a piece of music.
The time-span tree obtained as a result of GTTM analysis is used for melody morphing [6], melody prediction[7], and music summarization [8]. Therefore, we havedeveloped a manual editor for the GTTM and have analyzed and released 300 pieces of analyzed data [9].

Prolongation tree

Time-span tree

Metrical structureGrouping structure
Local grouping boundary

Figure 1. Grouping structure, metrical structure, timespan tree, and prolongational tree.
The previous database and analysis editor for timespan trees was limited, that is, the length of each piece ofmusic in the previous database was restricted to aroundeight bars because it was difficult for the analysis editor
to manipulate a piece with a large number of bars andnotes. The shortness of each piece in the GTTM databasecauses problems. For example, learning-based analyzers,
such as the σGTTM [4] or deepGTTM [5], only learn
with the data from eight bars of music, and the analysis
performance for pieces with more than eight bars suddenly worsens. As another example, we could not verify the
melody morphing method with a musical piece of morethan eight bars.
To solve the problems with the previous editor, wedeveloped a manual time-span tree generation tool that
supports a piece with a large number of bars and notes.
The experimental results show that the average editingtime with the time-span tree generation tool is shorter
than that with the previous tool. By using the generation
tool, four musicologists created 50 time-span trees forpieces of music with 32 bars over the course of half ayear.
This paper is organized as follows: Section 2 describes
related work, Section 3 discusses the problems of theprevious time-span tree editor, Sections 4 and 5 describe
an generation tool for editing time-span trees and the
GTTM database. Section 6 describes the experimental
results and Section 7 concludes with a summary and an
overview of future work.

2. RELATED WORK
Databases for music research have been created for various purposes [10–16]. The Real World Computing(RWC) Mu-sic Database is a copyright-cleared music
database that contains the audio signals and corresponding standard MIDI files for 315 musical pieces [10, 11].
The Repertoire International des Sources Musicales(RISM), an international non-profit organization with theaim of comprehensively documenting extant musicalsources from around the world, provides an online catalogue containing over 850,000 records, mostly for musicmanuscripts [12, 13]. The Variations 3 project provides
online access to streaming audio and scanned score images for the music community with a flexible access control
framework [14, 15]. The Essen folk song collection is adatabase for folk-music research that contains score dataon 20,000 songs, along with phrase segmentation information, and also provides software for processing thedata [16]. However, these research music databases [10–16] did not include deep musical structures like time-spantrees.
We previously proposed GroupingXML, MetricalXML, TimespanXML, and ProlongationalXML for
expressing deep musical structures, and we developed amanual editor for generating the structures [2, 9, 17] (Fig.
2). However, the manual editor for time-span trees had
the following problems:
 It was difficult for the editor to use a piece of musicof more than eight bars.
 Inputting the data of the structures needed a lot oftime in proportion to the exponent of the number ofnotes.
 The editor could not handle cadences.
Therefore, we have developed a tool for manually
generating time-span trees, which makes it possible todeal with longer pieces of music and input structures in ashorter amount of time, even for pieces of music with alarge number of notes. In addition, we have defined andindicated the position of cadences in the analysis data of
the TimespanXML.
Metrical StructureGrouping Structure
Piano roll score

3. PROBLEMS OF PREVIOUS TIMESPAN TREE MANUAL EDITOR
Analysis of a time-span tree requires a lot of time, and,
although the analysis time was shortened with the previous analysis editor, there were still the following problems: The score display was a piano roll. Figure 2 shows
the previous analysis editor for manual editing,which was difficult for musicologists to understandintuitively because the score display was a piano roll.
To solve this problem, in our new manual generation tool, described in Section 4.1, the score display
was changed to a staff score so that chord progression was easy for musicologists to grasp.
 It was difficult to manipulate a piece of music with alarge number of bars and notes. When the number ofbars in a piece increased, the width of the notes on
the piano roll became narrower, which made it difficult to understand each note’s value (Fig. 3 a).
When the number of notes in a piece increased, thenumber of branches in the time-span tree increased.
Therefore, it was difficult for us to visually distinguish between an essential stem and an orna-mented
branch in the time-span trees because stems andbranches from the stems were mostly straight lines(Fig. 3 b).
To solve these problems, as shown in Section 4.2,
we implemented the generation tool on a tablet device, which en-abled us to zoom in and out to visually understand the notes’ values and the numbers ofstems and branches.
(a) Note values on piano roll score

Time-span tree(b) Stems and branches

Prolongation tree

Figure 3. Difficulty of manipulating musical piece withlarge number of bars and notes.


Figure 2. Previous editor with piano roll score.

Top-down way of editing: In the first stage of timespan tree analysis, musicologists make a local tree
from the bottom up by using the results of thegrouping and metrical structure analyses. In ourprevious editor for manual editing, all linked timespan trees were given in advance and edited so thatthey became the target tree structure. Figure 4 shows
the previous editor; the position where the selectedbranch can be connected is highlighted in the timespan tree. Thus, if we moved the branch to the target
position, the branch would become highlighted andthen connect to the desired position.

4.3 Tree generation in bottom-up manner
Target position

Selected branch
Figure 4. Highlighting the position that can connect the
selecting branch in the previous editor for editing timespan tree.
The biggest problem with this method was that itwas not possible to do a single action regarding thesimplest and most frequently performed operations,such as swapping the parent and child of a branch.
For example, when we changed a time-span tree, as
shown in Figs. 5 a–c, the structure could not be directly changed because the time-span tree could bemade once Fig. 5 b. The method for editing thetime-span tree from the bottom up is described in
Section 4.3.

(a)

(c)

(b)

By simply placing a handle on another handle, a tree
structure can be generated in a way that the original handle becomes a sub branch and the handle that was placed
on top becomes the stem (main branch) (Fig. 6 b). After
combining a stem and a branch, it is possible to move
only the stem's handle. That is, the handle of the branchcannot be independently moved, and it will move following the movement of the stem. By clicking on and holding down the handle of the combined sub branch for along time, a menu comes up that gives the option of separating them again.
4.4 Cadential retention
In the time-span analysis of the GTTM, the positions of
the cadences in the time-span trees are indicated with anegg shape. However, the manual editor did not have sucha display function. The cadence was retained from thethird beat of the eighth bar in Fig. 6 c to the end of thepiece.
(a) Initial state after loading MusicXML

Step 2
Step 1(b) Tree generation in bottom-up manner
Figure 5. Manipulating time-span tree using previous
editor.

4. TIME-SPAN TREE GENERATIONTOOLThis section describes the four kinds of tool feature formanually generating time-span trees. The tool is datacompatible with the GTTM editor when using the MusicXML and the TimespanXML.

(c) Time-span tree with cadential retention

Cadential retention

4.1 Display staff notation on tablet
The tool for manually generating time-span trees loads
the MusicXML and displays it in staves (Fig. 6). By
zooming the tablet screen in and out, you can check boththe entire piece of music and individual sections. This
makes it easy to operate, even when the entire piece islong or when there are many notes.
4.2 Handles for tree operation
Since it is difficult to touch and select single notes orbranches with one’s fingertip, we added circular handles
to them. Handles were attached to all notes except forrests and grace notes. After loading the MusicXML , thehandles are seen directly above the notes (Fig. 6 a). The
heights of the handles are automatically separated because if two handles are very close to each other, it may
not be possible to select the desired handle.

Figure 6. Screenshot of manual time-span tree generation
tool.

5. GTTM DATABASE
Although we attempted to analyze polyphonic music, we
restricted the target analysis data to monophonic music in
the GTTM database because several rules in the theory
allow only monophony.

5.1 XML-based data structure
We used an XML format for all analysis data. MusicXML [18] was chosen as the primary input format
because it pro-vides a common ‘interlingua’ for music
notation, analysis, retrieval, and other applications. We
designed
GroupingXML,
MetricalXML,
andTimespanXML to express each structure. We also designed HarmonicXML to express chord progres-sions.
The XML format is suitable for expressing the hierarchical grouping structures, metrical structures, and timespan trees.
5.2 Score data
We collected 50 32-bar-long monophonic pieces of classical music that include notes, rests, slurs, accents, andarticula-tions entered manually with music notation software called Finale [19]. We exported the MusicXML byusing a plugin called Dolet. The 50 whole pieces and the
32 bars were selected by a musicologist (see appendix).
5.3 Analysis data
We asked four musicologists to manually analyze thescore data by faithfully using the GTTM. The time-spantree editor was used for analyzing grouping and metricalstructures, and the time-span tree generating tool wasused for analyzing time-span trees. The musicologists
also analyzed the chord progressions. Three other experts
then crosschecked the manually produced results.

6. EXPERIMENTAL RESULTS
We compared the time taken to input information into atime-span tree with the developed generation tool and theprevious editor. We asked three subjects who were music
beginners that had never used the time-span tree editor or
the generation tool before to create time-span trees withthe programs over the course of one hour each. The subjects input the desired information while looking at a
handwritten time-span tree on a printed staff score.
We prepared five pieces of music which were presented in a random order. There were a total of 10 trials
with the previous editor and the new generating tool for
creating time-span trees for these pieces, and the time foreach trial was recorded. Between the trials, the subjects
took a five-minute rest.
6.1 Construction of time-span trees from the starting
state
When the time-span tree generation tool first starts up, nobranches are connected (Fig. 6 a). On the other hand, inthe beginning state of the time-span tree editor, each noteis connected to a previous note as a branch (Fig. 7 a).
The trial results showed that the time for making
time-span trees was shorter when using the generation
tool for all pieces of music (Tab. 1). This is because thetime-span tree editor requires a long time to swap parent
and child branches.
6.2 Editing the output of the time-span tree analyzer
The start-up state was different between the editor andgeneration tool (see Section 5.1). In this section, we compare the time for editing time-span trees between the editor and the generation tool.
There is a time-span tree analyzer in the automatictime-span tree analyzer (ATTA) [2, 3] that has 13 adjustable parameters. The performance of the time-span tree
analyzers is different depending on the adjustable parameters. It took us an average of about 10 minutes per musical piece to find plausible tuning for the set of parameters.
Table 2 shows the F-measures of the time-span tree analyzer for each piece. The F-measure is given by theweighted harmonic mean of Precision P and Recall R.
We used the tree structure output by the ATTA as a starting state in this section.
The results show that the average editing time with thegeneration tool is shorter than that with the editor. However, the editor can edit faster than the generation tool inpieces with high F-measures, such as Moments Musicaux.
This is because, in a piece with a high F-measure, there
are few times in which switching between the stems andbranches is necessary and, as a result, it is possible toquickly edit with the editor. Conversely, the generation
tool requires a long time for switching between them because we need to join them again after disconnecting abranch with all the operations of the time-span tree.
Previous
editor [s]
589
737
906
985
923
757

1. Moments Musicaux
2. Wiegenlied
3. Traumerei
4. An die Freude
5. Barcarolle

Average (five pieces)

Developed
generation
tool [s]
361
390
513
588
521
474

Table 1. Construction of time-span tree from the startingstate.
Previous
editor [s]1. Moments Musicaux
2. Wiegenlied
3. Traumerei
4. An die Freude
5. Barcarolle

Average (five pieces)

124
524
758
645
345
479

Developed
generation
tool [s]
261
567
524
456
248
411

Fmeas
ure
0.84
0.69
0.63
0.48
0.60

Table 2. Editing the output of the time-span tree analyzer.

7. CONCLUSION
We have described the manual time-span tree generation
tool and the GTTM Database. The main contributions of
the study in this paper are as follows:

Development of a staff-based tool.
There have so far been only piano roll-based manual
editing tools, musicologists’ analyzed music written
by hand onto the staves of printed paper because
they required the staff for analysis. By developing atool for manually generating time-span trees, trial
and error became possible with the tool, which eliminated the need to print out sheet music.

Time-span tree generation in bottom-up manner.
Since the time-span tree generation tool generates
trees from the bottom up, we have been able to significantly shorten the editing time compared withconventional editing tools. Furthermore, with thezoom function, one can easily grasp the entire pieceor individual sections of long pieces of music.



Development of the GTTM database with pieces
around 32 bars.
Since the data analyzed so far was for eight-bar extracts of music, it was a big restriction when making
a learning-based analysis system or creating a system using time-span trees. Because the length wasincreased to 32 bars, it is expected that the performance of the analysis system will improve and thatthe applied system for time-span trees will expand.

Since we hope to contribute to the research of music
analysis, we will publicize our generation tool a datasetof a fifty pairs of a score and musicologists’ analysis results on our website:
http://gttm.jp/
We plan to analyze pieces other than classical pieces,such as jazz, in the future.
Acknowledgments
This work was supported by JSPS KAKENHI, Grant
Numbers 17H01847 and 16H01744.

8. REFERENCES[1] F. Lerdahl and R. Jackendoff, A Generative Theoryof Tonal Music, MIT Press, 1985.
[2] M. Hamanaka, K. Hirata, and S. Tojo, Implementing“A generative theory of tonal music,” Journal of
New Music Research, pp. 249–277, 2006.
[3] M. Hamanaka, K. Hirata, and S. Tojo, “ATTA:
Automatic Time-Span Tree Analyzer Based on
Extended GTTM,” Proc. of the 6th International
Conference on Music Information Retrieval
Conference (ISMIR 2005), pp. 358–365, 2005.
[4] M. Hamanaka, K. Hirata, and S. Tojo, “σGTTM III:
Learning-Based Time-Span Tree Generator Basedon PCFG,” Proc. of International Symposium on

Computer Music Multidisciplinary Research(CMMR2015), pp. 387–404, 2015.
[5] M. Hamanaka, K. Hirata, and S. Tojo, “deepGTTMI: Local Boundaries Analyzer Based on Deep
Learning Technique,” Proc. of International
Symposium on Computer Music Multidisciplinary
Research (CMMR2016), pp. 8–20, 2016.
[6] M. Hamanaka, K. Hirata, and S. Tojo, “Melody
Morphing Method Based on GTTM,” Proc. ofInternational
ComputerMusic
Conference(ICMC2008), pp. 155–158, 2008.
[7] M. Hamanaka, K. Hirata, and S. Tojo, “Melody
Expectation Method Based on GTTM and TPS,”
Proc. of the 9th International Conference on Music
Information Retrieval Conference (ISMIR2008), pp.
107–112, 2008.
[8] K. Hirata and S. Matsuda, “Interactive musicsummarization based on generative theory of tonal
music,” Journal of New Music Research, pp. 165–
177, 2003.
[9] M. Hamanaka, K. Hirata, and S. Tojo, “Musicalstructural analysis database based on GTTM,” Proc.
of the 15th International Conference on Music
Information Retrieval Conference (ISMIR2014), pp.
107–112, 2014.
[10] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka,“RWC Music Database: Popular, Classical, and Jazz
Music Databases,” Proc. of the 3rd International
Conference on Music Information Retrieval (ISMIR
2002), pp. 287–288, 2002.
[11] K. G. Saur Verlag, “RISM: International inventoryof musical sources, ” In Series A/II Music
manuscripts after 1600， 1997．
[12] M. Falletta, R. Hüsken, and K. Keil, “RISM:
academic and technical challenges of musicologicalsource research in an international framework,”
Studien und Materialien zur Musikwissenschaft,
Hildesheim, 2010.
[13] J. Riley, C. Hunter, C. Colvard, and A. Berry,“Definition of a FRBR-based Metadata Model for
the Indiana University Variations3 Project,”
http://www.dlib.indiana.edu/projects/variations3/doc
s/v3FRBRreportPhase2.pdf, 2008.
[14] J. Riley, “Moving from a Locally-Developed Data
Model to a Standard Conceptual Model,” Proc. of
the 10th International ISKO Conference, pp. 124–
130, 2008.
[15] H. Schaffrath, “The Essen associative code: A codefor folksong analysis,” In E. Selfridge-Field (Ed.),
Beyond MIDI: The Handbook of Musical Codes,
Chapter 24, pp. 343–361. MIT Press, Cambridge, 1997.
[16] M. Hamanaka, K. Hirata, and S. Tojo, “Interactive
GTTM Analyzer,” Proc. of the 10th International
Conference on Music Information Retrieval
Conference (ISMIR2009), pp. 291–296, 2009.
[17] MakeMusic
Inc.,“Finale,”2018,
http://www.finalemusic.com/.
[18] MakeMusic
Inc.,“musicXML,”2018,
http://www.musicxml.com/.

9. APPENDIX
List of musical piecesname and composer and bars that were used for analysis.
Name of Piece
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

Symphony No. 40 in G minor KV.550 1st mov.
Piano Concerto No. 3 in D minor 1st mov.
Cavalleria Rusticana "Intermezzo sinfonico"
Impromptus Op. 142, D. 935 No. 2 in A-flat major
String Quartet No. 4 Op. 18–4 in C minor
6 Pieces Op. 118 No. 2 "Intermezzo"
Myrtle of flower Op. 25 No. 1 "Dedication"
Scenes from Childhood Op. 15 No. 1 "Of Foreign Lands and Peoples"
Cello Sonata Op. 38 No. 1 1st mov.
4 Impromptus Op. 90 D. 899 No. 2 in E-flat major
Violin Sonata 1st mov.
The carnival of the animals No. 13 "The Swan"
Slavonic Dances 2 Op. 72 No. 2
Symphony No. 9 Op. 95 4th mov.
L’arlesienne 1st Suite No. 4 "Carillon"
Piano Trio No. 7 Op. 97 "Archduke" 3rd mov.
Symphony No. 9 Op. 95 2nd mov.
Serenade Lyrique
String Quartet Op. 18 No. 4 4th mov.
String Quartet Op. 64 No. 5 "Lark" 2nd mov.
Piano Sonata No. 16 in C major K. 545 1st mov.
Album for the Young Op. 68 No. 10 "Happy Farmer"
The Blue Danube Op. 314
The Nutcracker Suite Op. 71a No. 3 "Waltz of the Flowers"
9 Preludes Op. 1 No. 3
Myrtle of flower Op. 25 No. 7 "The Lotus-Flower"
Piano Quintet Op. 44 1st mov.
Four Impromptus D. 935 No. 3 "Ballade"
Piano Sonata No. 2 Op. 35
Swan Lake Op. 20a "Dance of the little swans"
The Seasons Op. 37a No. 10 "October-Autumn Song"The Seasons Op. 37a No. 11 "November-Troika"
Don Quixote "Grand pas de deux"
The Corsair "Variation"
O holy night
2 Melodies Op. 4 "Lydia"
Scenes from Childhood Op. 15 No. 2 "A strange story"
My old Kentucky home, good-night
Old Folks at Home
Piano Sonata No. 16 in C major K. 545 2nd mov.
Piano Sonata No. 16 in C major K. 545 3rd mov.
Piano Sonata No. 12 in F major K. 332 1st mov.
String Quartet Op. 3 No. 5 in F major 2nd mov.
Divertimento Hob. II–46 in B-flat major 2nd mov.
String Quartet No. 1 Op. 11 in D major 2nd mov.
La Gioconda "Dance of Hours"
6 Ecossaises No. 1 WoO. 83
25 Studies Op. 100 "Barcarolle"
Serenade No. 13 in G major K. 525 "Eine kleine Nachtmusik" 2nd mov.
Serenade for Strings in C major Op. 48 2nd mov. "Waltz"

Composer

MozartRachmaninoff
Mascagni
Schubert
BeethovenBrahmsSchumann
Schumann
BrahmsSchubert
FranckSaint SaensDvořákDvořákBizet
BeethovenDvořákElgar
Beethoven
Haydn
MozartSchumann
J. Strauss II
Tchaikovsky
Szymanowski
Schumann
Schumann
Brahms
ChopinTchaikovsky
Tchaikovsky
Tchaikovsky
Minkus
Adam
Adam
Faure
Schumann
FosterFosterMozartMozartMozartHaydn
Haydn
Tchaikovsky
Ponchielli
Beethoven
Burgmüller
MozartTschaikovsky

Bar

1–42
3–27
1–35
1–46
1–25
1–48
2–40
1–22
1–33
1–52
2–37
2–27
1–32
10–43
5–48
1–28
7–42
38–69
1–34
1–34
1–28
1–20
1–32
38–69
1–25
2–27
1–26
1–32
9–40
1–32
1–49
1–27
1–48
5–52
2–27
3–37
1–20
1–24
1–24
1–32
1–28
1–40
1–33
1–29
1–49
1–16
1–32
13–47
1–39
1–30

Measures

42
25
35
46
25
48
39
22
33
52
36
26
32
34
44
28
36
32
34
34
28
20
32
32
25
26
26
32
32
32
49
27
48
48
26
35
20
24
24
32
28
40
33
29
49
16
32
35
39
42