Computational Music Theory and its Applications to
Expressive Performance and Composition
Masatoshi Hamanaka1, Keiji Hirata2
and Satoshi Tojo3,
12

1-1-1, Tenodai, Tsukuba, Ibaraki, Japan, hamanaka@iit.tsukuba.ac.jp,
2-4, Hikaridai, Seikacho, Keihanna Science City, Kyoto, Japan, hirata@brl.ntt.co.jp,
31-1, Asahidai, Nomi, Ishikawa, Japan, Tojo@jaist.ac.jp

Abstract. This chapter describes a music analysis system based on a generative
theory of tonal music (GTTM). Musical theory provides methodologies foranalyzing and transcribing musical knowledge, experiences, and skills from amusician's way of thinking. Our concern is whether the concepts necessary formusic analysis are sufficiently externalized in musical theory. Given its ability
to formalize musical knowledge, we consider the GTTM to be the mostpromising theory among the many that have been proposed because it captures
the aspects of musical phenomena based on the Gestalt in the music and follows
relatively rigid rules. It also describes music expectation and melody morphing
methods that use the analysis results from our music analysis system. The
music expectation method predicts the next notes needed to assist musicalnovices in playing improvisations. The melody morphing method generates anintermediate melody between two melodies in a systematic order in accordancewith a certain numerical measure.
Keywords: computational music theory, generative theory of tonal music(GTTM), melody expectation, melody morphing, tonal pitch space (TPS), timespan tree.

1

Introduction

Our goal is to create a system that will enable a musical novice to manipulate a pieceof music, which is an ambiguous and subjective media, according to his or her
intentions. We believe that it is important to make it possible to create a musicalsystem for musical novices that can:1) easily manipulate a piece of music, and2) mirror the user's intentions.
Note that the higher the abstraction level of the objects for manipulating the music is,
the more difficult it becomes to reflect the user's intentions. For example, it is difficult
for musical novices to manipulate music with commercial music sequence softwarethat only operates on the surface structure of music, that is, the pitch and note-ontiming of each note. On the other hand, Garageband [1] can create a piece of music
though simple manipulations, i.e., by just concatenating pre-stored phrases. However,when we want arrange a portion of a melody in a phrase, we have to manipulate thesurface structure of the music, and a musical novice would find it difficult for thesoftware to mirror his or her intentions in such a case.
Music theory, in particular, focuses on a piece written on a score, gives us themethodology for analyzing and understanding a piece, and explains deep structure,musical knowledge, experiences, and skills in a comprehensive way. We believe that
implementing music theory on a computer would bring great benefits because it could
provide a theoretical basis for developing a support system for musical activities. For
instance, such advantages include performance rendering [2-4] and musicsummarization for practical use when displaying the results of a music information
retrieval system [5].
We have developed a system of music analysis called ATTA [6, 7] and FATTA [8]based on a music theory called a generative theory of tonal music (GTTM) [9]. ATTA
and FATTA can generate a time-span tree as the result of a GTTM analysis. The
GTTM consistently represents multiple aspects of music in a single framework. Thisfeature is important when the musical system is to assist a musical novice inmanipulating musical structures. For instance, if we imagine a simple operation thatsplits a melody, the split operations may vary depending on the relevant musicalstructure. Therefore, it is preferred that the splitting position of the melody and that of
the ornamented melody be identical. Unless a system of consistent operations in termsof melody, rhythm, and harmony is developed, the resulting splitting position may bedifferent.
With regard to how the software should reflect the user's intentions, we take an
example of composing a melody, where it is supposed that a user wants to arrange
melody A by adding some musical nuances to it and he/she knows melody B has suchnuances. If a user could use a system accepting a morphing command, issuing asimple command, for example, "add the nuance of melody B to melody A", canaccurately convey a user's intention to a morphing system. As a result, the systemgenerates multiple melodies approaching from melody A to B by little and little. The
advantage of morphing is not only its simple and accurate transferability andmanipulability but also its ease of understanding the relationship between inputs andoutputs.
Previous music systems [10-12] have their own way of music analysis, from which
it is difficult to acquire deeper musical structures, and thus these systems are difficult
to manipulate according to the user's intention. On the other hand, Hirata [13] defineda representation method and primitive operation for polyphony, and this development
indicates the potential for constructing a melody arranging algorithm.
We developed a melody morphing method in which we input monophonies A and
B and then generate intermediate melodies between melodies A and B with asystematic order according to a certain numerical measure by configuring theparameters that determine the level of influence of the features of melodies A and B[14, 15]. Our system exploits melody morphing to make it possible for novices tocreate melodies reflecting their intensions.
Melody prediction is one of the most difficult problems in musical information
retrieval because composers and players may or may not create melodies that conformto our expectation. The development of a melody expectation method is thus
important for building a system that supports musical novices because melody
expectation is one of the most basic skills for a musician.
Our melody prediction method helps a novice construct a melody or play an
improvisation by displaying candidates for the next notes. This method is designed to
be used with a “expectation piano” (Fig. 1), which was developed to assist musicalnovices play improvisations. On the lid of this piano, there is a 32 × 25 full-color
LED matrix that displays a piano roll view that scrolls down in time with the music.

Fig.1. Expectation Piano.
We identified two key requirements for our melody expectation method to make it
useful to musical novices playing an improvisation on the expectation piano.
1) Candidate notes are predicted and output even if the input melody is novel.
2) The output is appropriate from a musical point of view.
Two approaches were considered when developing this method: statistical learningand music theory. With the statistical learning approach, the predictions depend on
the characteristics of the data used for learning: composer, genre, period, country, etc.
[16, 17]. Moreover, predicting candidate notes for a novel melody is problematic
because the system may not be able to find a similar melody in the learning data andthus may be unable to evaluate whether the notes are appropriate or not. With themusic theory approach, the predictions do not depend on the characteristics of thedata used for learning. It can thus be applied to novel melodies.

Unlike most previous prediction methods, which use statistical learning, ourmethod evaluates the appropriateness of each candidate note from the view point ofmusical theory. Although many music theories have been proposed [18–21], GTTM isthe most suitable for predicting notes in a melody because it can be used to represent
the various aspects of music in a single framework. In particular, our prediction
method uses the concept of melody stability based on the GTTM and the tonal pitch
space (TPS) [22] to evaluate the appropriateness of the melody. It can thus predict thecandidate next notes not only from the surface structure of the melody but also fromthe deeper structure of the melody acquired by GTTM and TPS analysis.
The organization of this chapter is as follows. In Section 2, we briefly describe
music theory GTTM, discuss the problems and how to solve them when
implementing GTTM, and propose exGTTM as extension of GTTM. In Section 3, we
describe an interactive analyzer for GTTM, which enables seamless change of theautomatic analysis process with an ATTA and the manual edit process with a GTTM
manual editor. In Section 4, we present the melody expectation method that predicts
the next notes is described for assisting musical novices to play improvisations. In
Section 5, we explain the melody morphing method that generates an intermediate
melody between a melody and another melody with a systematic order according to acertain numerical measure. In Section 6, we evaluate the performance of theimplemented system. Finally in Section 7, we conclude with a summary and overview
of future work.

2

Implementing GTTM on a computer

GTTM is a music theory for describing the insight of an “experienced listener,” and it
consists of four sub-theories: grouping-structure analysis, metrical-structure analysis,time-span reduction, and prolongational reduction.
The grouping-structure analysis hierarchically divides a series of notes of a
homophony into phrases or motives. It seems that a singer looks for breathing points
when singing a long melody(Fig. 2). The metrical-structure analysis identifies strong
and weak beats at each metrical level: quarter note, half note, whole note, twomeasures, and four measures. Essentially, it looks for timings at which a listener beats
time with his/her hands to music or a conductor swings his/her baton. Time-span
reduction distinguishes important parts of a melody from unimportant ones and yields
a binary tree, called a time-span tree, so that each structurally important note belongs
to a stem at every level (Fig 3). The prolongational reduction generates a tree
structure representing subordinate relationships between chords by explicitly
indicating harmonic retention and change.
Each sub-theory of GTTM is described by two kinds of rules: a well-formedness
rule prescribes the conditions and constraints that must always be satisfied; apreference rule prescribes which structure is preferable among the structuressatisfying the well-formedness rules. Thus, depending on the situation, some
preference rules hold and others do not.
Among the four sub-theories, this section presents the methodology for
implementing the grouping-structure analysis, metrical-structure analysis, and timespan reduction [6-8, 23, 24]. Since the prolongational reduction is still evolving andcontroversial, we do not implement it at present.

Time-span Tree

Metrical StructureGrouping Structure
Fig. 2. Grouping structure, metrical structure, and time-span tree. Groups are graphicallypresented as several levels of arcs below a music staff. Strong beats are illustrated by dots in
multiple levels below the music staff.

Fig. 3. Left side shows a melody and its corresponding time-span tree, where a single
note, called a head (note C4 shown in right side), represents the time span denoted by<---> containing the two notes.
2.1

Problems in implementing GTTM

when we make GTTM rules operate on a computer, we have to make an ambiguous
concept a firm definition and supplement the lack of concepts. Roughly speaking,these tasks correspond to parameterization and externalization, respectively. Here, we
discuss the problems occurring in the process of parameterization and externalization.
Ambiguous concepts defining preference rules. GTTM uses some undefined words
that can cause ambiguities in the analysis. For example, GTTM has rules for selecting
structures for discovering similar melodies (called parallelism), but does not have thedefinition of similarity itself. To solve this problem, we attempted to formalize thecriteria for deciding whether each rule is applicable or not.
Conflict between preference rules. Conflict between rules often occurs and resultsin ambiguities in the analysis because there is no strict order for applying thepreference rules. Fig. 4 shows a simple example of a conflict between grouping
preference rules (GPR). To solve this problem, we introduced adjustable parametersthat enable us to control the strength of each rule.

Fig. 4. GPR3a (register) is applied between notes 3 and 4 and GPR6 (parallelism) is
applied between notes 4 and 5. A boundary cannot be perceived at both 3-4 and 4-5,
because GPR1 (alternative form) strongly prefers that note 4, by itself, cannot form agroup.

Lack of working algorithm. Knowledge represented in the rule form is in generaldeclarative, which is advantageous in the sense that a knowledge programmer does
not need to take into account an algorithm for reasoning. A system is required toperform automatic reasoning on the declaratively described knowledge. Unfortunately,
GTTM has few descriptions of the reasoning and working algorithms needed tocompute analysis results.
2.2

Solution: proposal of exGTTM

We extended the GTTM theory through full externalization and parameterization anddevised a machine-executable extension of GTTM, exGTTM. The externalization
includes introducing an algorithm for generating a hierarchical structure of the timespan tree in a mixed top-down/bottom-up manner. Such an algorithm has notpreviously been represented for GTTM. The parameterization includes a parameterfor controlling the priorities of rules to avoid conflicts among them as well as
parameters for controlling the shape of the hierarchical time-span tree. Although it has
been suggested that such parameters are required in GTTM, they were not explicitly
presented.
Here, we distinguish two kinds of ambiguity in music analysis: one involves themusical understanding by humans, and the other concerns the representation of amusic theory. The former kind of ambiguity derives from the ambiguity of the music
itself. For the latter type of ambiguity, related to GTTM, either no concept formechanization has been presented, or it has only been presented in an implicit way.
Therefore, due to the former kind of ambiguity, we assume there is more than one
correct result. We avoid the latter kind of ambiguity as much as possible byperforming full externalization and parameterization.
Full externalization and parameterization. The significance of full externalizationand parameterization is twofold: precise controllability and coverage of the manual
results. Whenever we find a correct result that exGTTM cannot generate, we
introduce new parameters and give them appropriate values so that it can generate thecorrect result. In this way, we repeatedly externalize and introduce new parameters
until we can obtain all of the results that people consider correct. In total, we
introduced 15 parameters for grouping-structure analysis(Table 1), 18 for metricalstructure analysis(Table 2), and 13 for time-span reduction (Table 3).
We appropriately supply lacking parameters and make implicit parameters
explicit 1. The parameters introduced by exGTTM are categorized into identified,
implied, and unaware.
A parameter in the first category is identified in GTTM but it is not assigned
concrete values. Hence, we valuate such a parameter. For example, since the resulting
value of the GPR2a application, DGPR2a, is binary, if the rule holds, DGPR2a makes 1,
and 0 if it does not hold. On the other hand, since GPR6 holds indefinitely, theresulting value of GPR6, DGPR6, varies continuously between 0 and 1.
1

In the chapter, the word “parameter” is used not only for parameters used in controlling asystem externally but also for internal variables (intermediated variables) connecting submodules.

A parameter of the second category is implied in GTTM. Hence, we make itexplicit. For example, to resolve the preference rule conflict, we introduce parameters
to express the priority for each preference rule (SGPR R, SMPR R, and STSRPR R in Table 1,
2 and 3). Since each preference rule has its own priority, all of the priority patterns are
realized. This is an example of full-parameterization.
For the third category, we need to complement parameters that are not recognizedin the original theory, since some of them may nearly lack any musicological meaning.
For example, GPR6 in exGTTM needs to add parameters for controlling theproperties of parallel segments, including the weights for pitch-oriented matching or
timing-oriented matching.
We add a comment to the domain of intermediate variables, denoted as D and B.
The domain of all the intermediate variables is constrained within the range of 0 to 1,
and for this purpose, these variables are normalized at every computing stage. Thanks
to this property, exGTTM can flexibly combine any intermediate variables (andpossibly parameters) and cascade as many weighted-mean calculations as needed.
This facilitates precise controllability.
Table 1. Fifteen adjustable parameters for grouping-structure analyzer.
Parameters
Description
Strengthof each grouping preference rule. The larger the
SGPR R (0≦SGPR R≦1)
value is, the stronger the rule acts. R∈｛ 2a, 2b, 3a, 3b,3c, 3d, 4, 5, and 6 ｝
Standard deviation of a Gaussian distribution, the averageσ (0≦ σ ≦0.1)
of which is the boundary by GPR5. The larger the valueis, the wider its skirt becomes.
Balance between temporal similarity of attack points andWm (0≦ Wm ≦1)
that of pitch difference in GPR6. The larger the value is,the more the system estimates the pitch difference.
Weight for the length of parallel phrases. The larger theWl (0≦ Wl ≦1)
values is, the more the length of parallel phrases isprioritized in GPR6.
Balance determining whether the note i becomes theWs (0≦ Ws ≦1)
ending note of a group or the beginning note of thefollowing group in GPR6. The larger the value is, themore the note tends to be the ending note.
Threshold at which the effects of GPR2,3 are considered
TGPR4 (0≦ TGPR4 ≦1)
to be salient in GPR4. The smaller the value is, the moreprobably GPR4 is applied.
Threshold in the lower-level boundary. The smaller the
Tlow (0≦ Tlow ≦1)
value is, the more salient the boundary becomes.

Table 2. Eighteen adjustable parameters for metrical-structure analyzer.
Parameters
SMPR R(0≦SMPR R≦1)

Wm

(0≦ Wm ≦1)

Wl(0≦ Wl ≦1)

Ws(0≦ Ws ≦1)

TMPR R(0≦ TMPR R ≦1)

Description
Strength of each metrical preference rule. The larger
the value is, the stronger the rule acts.
R∈｛1,2,3,4,5a, 5b, 5c, 5d, 5e, and 10)
Balance between temporal similarity of attack points
and that of pitch difference in MPR1. The larger thevalue is, the more the system estimates the pitchdifference.
Weight for the length of parallel phrases. The larger
the value is, the more the length of parallel phrases isprioritized in MPR1.
Balance determining whether the note i becomes theending note of a group or the beginning note of thefollowing group in MPR1. The larger the value is, themore the note tends to be the ending note.
Value of the threshold that decides whether each rule
is applicable. R∈｛ 4, 5a, 5b, 5c ｝

Table 3. Thirteen adjustable parameters for time-span tree analyzer.
Parameters
STSRPR R(0≦STSRPR R≦1)

Wm

(0≦ Wm ≦1)

Wl(0≦ Wl ≦1)

Ws(0≦ Ws ≦1)

Description
Strength of each rule. The larger the value is, thestronger the rule acts.
R∈ {1, 2, 3, 4, 5a, 5b,5c, 5d, 5e, and 10}
The balance between the temporal similarity of attack
points and that of the pitch difference in TSRPR4.
The larger the value is, the more the system estimates
the pitch difference.
The weight for the length of parallel phrases. The
larger the values is, the more the length of parallelphrases is estimated in TSRPR4.
The balance determines whether the note i becomesthe ending note of a group or the beginning note of
the following group in TSRPR4. The larger the valueis, the more the note tends to be the ending note.

Algorithm for acquiring hierarchy. Among the issues that require working
algorithms, the problems for acquiring hierarchical structures in the grouping- andmetrical-structure analyses and the time-span tree reduction can be all regarded as
constraint satisfaction problems (CSP). This is because only the properties to besatisfied for the hierarchical structures are represented in the form of a rule; that is,neither constraint nor order of generating hierarchical structures is determined inadvance.
The constraints stipulated by the GTTM rules are divided into two categories: local
and global. The former includes GPR2 (proximity) and TSRPR1 (strong metrical
position), and the latter GPR5 (symmetry) and MPR1 (parallelism). We need tohandle global constraints carefully when generating hierarchical structures. For theexample of GPR5 in Fig. 5, given a group at Layer 1, an inner boundary likely occurs
around the center of the group, that is, either between Notes 1 and 2 or between Notes
2 and 3. Here, we can consider two cases. In Case 1, the boundary between Notes 1
and 2 is selected, taking into account the effects of some other rules. Then in eachsubgroup in Layer 2, the inner boundary of the subgroup may occur on the left-handside of a center note. On the other hand, in Case 2, the boundary between Notes 2 and3 is selected. Therefore, the inner boundary may occur on the right-hand side of acenter note.
In light of the above considerations, we are developing algorithms for generating
hierarchical structures for exGTTM so that nodes are generated either from thebottom-most nodes or the top-most node incrementally and so that every time thenodes at a layer are calculated, global information is re-calculated before moving onto
an adjacent layer.

Group

♪

Note 1

2 3

♪♪♪

♪

Layer 1
Boundary around the center

Case 1: Boundary between Notes 1 & 2 selected
Subgroup

♪

♪♪♪

♪

Layer 2
Case 2: Boundary between Notes 2 & 3 selected
Subgroup

♪

♪♪♪

♪

Layer 2
Fig. 5. In computing GPR5, the determined boundary position influences theidentifications of remote boundaries in lower layers, and we have to take into account
up-to-date global information every time. That is, a global constraint is inevitably
dynamic.

2.3

FATTA: Fully Automatic Time-span Tree Analyzer

We implemented a time-span tree analyzer, called automatic time-span analyzer(ATTA), based on exGTTM. Although ATTA can automatically acquire a time-spantree, because the parameters are manually controlled, it takes too much time to find aset of optimal parameters. Therefore, we developed a method for automaticallyestimating the optimal parameters [8].
Two rules in GTTM [9] are not implemented in ATTA: GPR7 and TSRPR5.
GPR7 (time-span and prolongational stability): prefer a grouping structurethat results in a more stable time-span and/or prolongational reductions.
TSRPR5 (metrical stability) In choosing the head of time-span T, prefer achoice that results in a more stable choice of metrical structure.
These rules require that information from later processes, such as timespan/prolongational reductions, be sent back to earlier processes.
To estimate the optimal parameters automatically, we evaluate the structural
stability of the analysis results derived by ATTA. We use GPR7 and TSRPR5 to
calculate the level of stability. Fig. 6 shows the process flow of the FATTA, which
consist of the ATTA and a loop by the GPR7 and TSRPR5.
Implementation of GPR7 with Tonal Pitch Space. GPR7 is applied to the loop
between the time-span/prolongational reduction and grouping structure analysis. This
rule leads to a preference for a grouping structure that results in a more stable timespan and/or prolongational reductions. The holding level of GPR7, which varies
continuously between 0 and 1, is defined as∑ distance( p(i), s(i)) × size(i) .
=∑ size(i)2

DGPR 7

(1)

i

2

i

where i indicates the head of the time-span, which has primary and secondarybranches, denoted by p(i) and s(i), respectively. Distance (x, y) indicates the distance
between notes x and y in the tonality of the piece, which are defined using Lerdahl’s
tonal pitch space [2]. We normalized the distance from 0 to 1. The size(i) indicates thelength of the time-span with head i. When calculating DGPR7, we use the square ofsize(i) for the weighting for empirical reasons.
In the tonal pitch space, the distance between chord x = C1/R1 and chord y =C2/R2 is defined as follows:δ ( x → y ) = i + j + k ,.

(2)

where i is region distance, j is chord distance, and k is basic space difference. The
region distance is the smallest number of steps along the regional circle of fifth between
R1 and R2. The chord distance is the smallest number of steps along the chordal circleof fifth between the roots of C1 and C2 within each region. The basic space distance is a

specially weighted to define each chord and region. Note that pitch class only has ameaning in terms the elements of the sets that define chords and regions, and chords arealways understood as functioning within some region.
Implementation of TSRPR5. TSRPR5 is applied to the loop between the time-span
reduction and metrical structure analyzer and results in a more stable metricalstructure when choosing the head of a time-span. The holding level of TSRPR5,
which varies continuously between 0 and 1, is defined as

DTSRPR 5 =size(i ) 2
1×∑∑ size(i)2 i  0

dot ( p(i )) ≥ dot ( s (i )) ,.
dot ( p (i )) < dot ( s (i ))(3)iwhere dot(x) indicates the number of metrical dots of note x.
Optimization of adjustable parameters. The set of optimal ATTA parameters is
obtained by maximizing the average of DGPR7 (0 ≤ D GPR7 ≤ 1) and DTSRPR5 (0 ≤DTSRPR5 ≤1). The parameters and default values are Srules = 0.5, Trules = 0.5, Ws = 0.5,
Wr = 0.5, Wl = 0.5, and σ = 0.05. Because there are 46 parameters, a great amount oftime is needed to calculate all parameter combinations. To minimize the calculation
time, we constructed an algorithm that does the following:(1) Maximize average of DGPR7 and DTSRPR5 by changing a parameter from its
minimum to its maximum value.
(2) Repeat (1) for all parameters.
(3) Iterate (1) and (2) as long as the average of DGPR7 and DTSRPR5 is higher than after theprevious iteration.
FATTA
Feedback loop
optimization

optimization
Adjustable parameters forgrouping structure analyzer

optimization

Adjustable parameters for
metrical structure analyzer

Adjustable parameters fortime-span tree analyzer

ATTA

Applying TSRPR5
Applying GPR7

DGPR 7DTSRPR5

Level of time-span tree stability

Fig. 6. Processing flow of fully automatic time-span tree analyzer (FATTA).

3

Interactive GTTM Analyzer

Fig. 7 is a screenshot of the viewer of our interactive GTTM analyzer. There is asequence of notes displayed in a piano roll format. Below the notes is a grouping
structure, which is graphically presented as several levels of arcs. Below the grouping
structure is a metrical structure. Strong beats are illustrated as several levels of bars.
Above the notes, there is a time-span tree. Below the time-span tree is a
prolongational tree, a binary tree that expresses the structure of tension and relaxation
in a piece of music.

Prolongational
Tree
Time-span
TreeGrouping
Structure
Metrical
Structure

Fig. 7. Screenshot of interactive GTTM analyzer.
Fig. 8 is an overview of our interactive GTTM analyzer, consisting of an ATTA[6,7], a GTTM manual editor, and a GTTM process editor. The ATTA consists of agrouping structure, a metrical structure, and time-span tree analyzers. We have beendeveloping a prolongation-al tree analyzer.

ATTA

GTTM process editor

Grouping structure analyzer
Metrical structure analyzer
Time-span tree analyzer
Prolongational tree analyzer

GTTM manual editor

Grouping structure editor
Metrical structure editor
Time-span tree editor
Prolongational tree editor
Tonal Pitch Space editor

Fig. 8. Overview of interactive GTTM analyzer.

The GTTM manual editor consists of grouping, metrical, time-span, prolongational,
and Tonal Pitch Space editors. The Tonal Pitch Space [22] is a music theory for chord
progression composed by Lerdhal, who is one of the authors of the GTTM. Although
the GTTM includes rules that require the analysis results of chord progression, the
ATTA uses such rules by adopting the results of the Tonal Pitch Space.
The analyzing process with the ATTA and GTTM manual editor is complicated,and sometimes a user is confused as to what he or she should do in the next process,as there are three analyzing processes in the ATTA and five editing processes in the
GTTM manual editor. A user may iterate the ATTA and manual edit processes
multiple times. To solve this problem, we propose a GTTM process editor, which
presents candidates for the next process of analysis, and a user only needs to change
the process, just by selecting the next process.
We use an XML format for all the input and output data structures of ourinteractive GTTM analyzer. Each analyzer and editor of our analyzer works
independently, but they are integrated with the XML-based data structure.

3.1

GTTM Manual Editor

In some cases, the ATTA may produce a preferable result, which reflects the user’s
interpretation, but in other case it may not. When a user wants to change the analysis
result according to his or her interpretation, he or she can use the GTTM manual
editor. We describe the method for editing and constructing a musical structure of the
GTTM using the GTTM manual editor.
Grouping structure editor. Fig.9 is a screenshot of our interactive GTTM analyzer
in editing a grouping structure. The color of the target group and all its subgroups turn
red after selection with a mouse. Then we can open a popup menu by right clicking
the mouse. There are four operations in the popup menu, divide this group and create
subgroup, divide this group, delete, and delete descendant.
To change a position of a grouping boundary, a user first delete the groups which
adjoin the boundary then divide the upper level (global level) group and create newsubgroups where he or she wants to create a boundary. By left clicking a groupingboundary, the user sees the rules that are applied to the boundary and he or she can
add or delete these rules.

Fig. 9. Screenshot of grouping structure editor.

Metrical structure editor. Although the metrical structure analyzer in the ATTA
performs fairly well [24], a user may want to slightly edit the metrical structure. In
which case, he or she applies the metrical structure editor, and changes the strength
level of a beat by dragging a bar up or down. At the same time, he or she sees therules that are applied to the bar and can add or delete these rules.
While editing beat strength, a user may break hierarchical metrical structures. In
other words, the results of the metrical structure editor sometimes do not hold for the
metrical preference rules. This problem can be solved using the GTTM process editor,
which we discuss in 3.2.
Time-span tree editor. In the time-span tree, each branch has a head represented by asquare in the time-span tree editor, and a user can move the head by dragging anotherbranch. Fig. 10 is a screenshot of dragging a head. The light blue branch is the former
position, and the dark blue branch is the latter position. A user can select a type foreach head by opening the popup menu among those four types, ordinary, fusion,transformation, and cadential retention.

Fig. 10. Screenshot of when dragging head.

Prolongational tree editor. The process of the prolongational tree editor is the sameas that for the time-span tree. The prolongational tree is constructed by reconnecting
the heads based on the time-span tree. There are head connection constraints of the
prolongational tree. When a head connection of a prolongational tree is ill -formed,
the GTTM process editor automatically opens the popup menu and displays
candidates for a solution.
Tonal Pitch Space editor. The reason we include a Tonal Pitch Space editor in ourinteractive GTTM analyzer is that the editor provides quantitative grounds for the
prolongational tree to be hierarchical. Therefore, analyzing the Tonal Pitch Spacewith the prolongational tree improves analyzing performance.
3.2

GTTM Process Editor

In the GTTM, the analysis sequence proceeds from the grouping structure, secondly
to the metrical structure, next to the time-span tree, and finally to the prolongational
tree. However, the GTTM contains feedback links from higher- to lower-levelstructures. For example, grouping preference rule 7 (GPR7) (time-span andprolongational stability) prefers a grouping structure that results in a more stable timespan and/or prolongation reduction. Therefore, to analyze with feedback link rules,
we need to perform several analyzing processes by trial and error. The GTTM process
editor helps in this repetition by performing three functions, data inputting, history
recording, and process controlling.
Data inputting. Data inputting helps with the input of analysis results, which areprepared by another user or analyzer. For example, we do not have an automatic
analyzer for the Tonal Pitch Space [22] in our interactive GTTM analyzer; however,attempts have been made to implement the Tonal Pitch Space, so we can use thoseresults [25].
We can add new rules to the ATTA using data inputting. For example, grouping
preference rule 6 (GPR6) is a rule for parallelism in a grouping structure; however,the GTTM does not define the decision criteria for construing whether two or moresegments are parallel or not. Therefore, many implementations of GPR6 would bepossible, although we propose only one of them. By adding a new rule to the ATTA,
we can control a new adjustable parameter for the new rule, GPR6+, which is the new
implementation of GPR6.
History recording. History recording records the operation of analysis, and a user
can return to the previous phase of analysis. History recording enables the copying
and pasting of several operations of analysis while editing parallel phrases.
In the GTTM, there are few descriptions of the reasoning and working algorithms
needed to compute the analysis results, especially for the time-span andprolongational trees. By using history recording, we look forward to storing theanalysis knowledge, which improves automatic analysis.

Process controlling. Process controlling enables seamless change of the analysis
process by using the ATTA and the manual edit process by using the GTTM manual
editor, representing candidates for the next process of analysis. The representation
method differs depending on the number of candidates for the next process.
When there is only one candidate process, the process-controlling function
automatically executes the process. For example, when a user edits the strongest beatin Fig. 11a in the 2nd level, the hierarchical metrical structure is broken because inlevel 3 of Fig. 11b there are three weak continuous beats, and the metrical wellformedness rule 2 (MWFW2) does not hold. MWFR2 requires that strong beats arespaced either two of three beats apart at each metrical level. The process editor
automatically alternately produces strong and weak beats in level 3 (Fig. 11c). If there
is a higher metrical structure than level 3, the metrical analyzer of the ATTA
automatically analyzes after level 3 and constructs a hierarchical metrical structurereflecting the user’s intention.

(a) Original structure
level

4

Strongestbeet(b) Structure broken
by user editing
User editing(c) Automaticallysolve using
process controller

32
1
Three weak continuous beatFig. 11. Automatically correct broken metrical structure.
When there are a few candidates, the process controlling function automatically
opens the popup menu and shows the candidates. For example, if there is a grouping
structure, as shown Fig. 11a, and a user deletes a group at the upper left (Fig. 12b),
the grouping structure of Fig. 12b is broken since grouping well-formedness rule 3(GWFR3) does not hold. GWFR3 requires constraints that a group may containsmaller groups. To solve this problem, there are only two processes:
1) Delete all the groups at the same level of the deleted group (Fig. 12c).
2) Extend the grouping boundary of the left end of the right group of thedeleted group to the left end of that deleted group (Fig. 12d).
The next process can be executed by selecting one of the two processes displayed in
the popup menu.

(a) Original structure(b) Structure broken
by user editing(c) Solution 1(d) Solution 2: Deleted group

Fig. 12 Two types of solutions for broken grouping structure.
When there are many candidates, the process-controlling function selects andshows the top-ten candidates from the history recording. The candidates are ordereddepending on the similarity of the history. For example, after editing a time-span treewith the time-span tree editor, executing a grouping analyzer or metrical analyzer inthe ATTA is ranked in the upper levels because there are rules for feedback link suchas GPR7 or metrical preference rule 9 (MPR9). GPR7 (time-span and prolongational
stability) is a link from the time-span and prolongational trees to the grouping
structure, and MPR9 (time-span interaction) is a link from the time-span tree to themetrical structure.
We have not implemented the original ATTA on GPR7 and MPR9. In this section,
we omit the details of the implementation of these rules due to space limitations.

4

Melody Expectation

Our melody expectation method predicts candidate notes by using the level ofstability of the time-span tree defined in FATTA. Our position is that we cannot
always specify a single expected, following tone; and thus, we developed an
expectation piano that simply suggests multiple candidates among stable pitch
events with higher stability. The functions of FATTA are restricted as follows;
FATTA only treats monophonic western tonal music. Thus, our expectation method
can predict only monophonic musical structures of western tonal music.
4.1

Melody Expectation Method

The main advantage of our melody expectation method is that, the stability of a
melody is calculated by analyzing the whole melody from the beginning note to the
expected note, not from only the local melody(a few notes previous to a relevantnote) ; previous melody expectation methods based on music theory (eg. Steve
Larson's theory of musical forces [21]) derive the expected note from the local
melody. Music tends to be more interesting when it does not match the listener’s
expectations, such as a delayed note, and this may result in tension and relaxation. A
composer can deliberately construct such music, which can make it difficult to predict
the next notes in the melody with accuracy. For example, an ornamentation note isoften inserted before the expected note. In such cases, our method can predictcandidate notes fairly well because FATTA can evaluate the stability of the entire
structure of the time-span trees, which includes branches connected to essential notes
and leaves connected to ornament notes.
Real-time Extension for FATTA. To be able to predict notes on the basis of GTTM,
FATTA must run in real time. However, FATTA needs several minutes to finish theanalysis, so running in real time is difficult. Therefore, we extended the algorithm toenable real-time operation. First, to speed up the iteration described in 2.2, we use theset of optimal parameter values for the last melody, which is one note shorter than thepresent melody, as the initial parameter set. Second, to reduce the time used by ATTA,
we introduce a window for analysis. The size of the window is the longest group
length within 16 measures of the present position. This length can be acquired
through preprocessing using the grouping structure analyzer in ATTA. If there is nogrouping boundary in 16 measures from the present position, we use 16 measures as
the window size.

Calculation Level of Stability of Melodies by FATTA. Our method evaluates theappropriateness of each candidate melody to occur after the present one by calculating
its stability. We use the average of DGPR7 and DTSRPR5 as the level of stability acquiredby FATTA.
Fig. 13 shows the calculated level of stability from the primary note to the present
note. The level of stability can be calculated after the third note because GTTM
analysis needs at least four notes.

Level ofstability

1.0
Time

0
Fig. 13. Level of stability over time.

In the score, the primary note is a tonic of the region, and the tail note is also a tonicof the region. This means that the levels of stability indicate a large value at the tail ofthe melody and a smaller value in the middle of the melody. Therefore, the higher thelevel of stability, the relatively closer the tonic of the region. The region of the
melody and chord progression were estimated in the implementation of GPR7 by
applying the tonal pitch space.
4.2

Expectation Piano

Our expectation piano assists novices with musical improvisation by displaying thepredicted notes on the piano lid. When the novice finds it hard to continue playing the
melody, she/he can continue the improvisation by playing a note displayed on the lid,
without impairing tonality.
Overview. The processing flow of the prediction piano is as follows. First, the MIDI
signals for the music played on the piano are sent to a computer. The signals arequantized, and a MusicXML version of the performed melody is created. A learningbased quantization method [26] is used to eliminate the deviation in onset times,
which are then aligned to the normalized positions. The predicted notes are acquiredby inputting the MusicXML into FATTA. Finally, the predicted notes are displayedon the piano lid.

LED Scrolling Piano Roll. Our method evaluates the appropriateness of eachcandidate melody to occur after the present one by calculating its stability. We use theaverage of DGPR7 and DTSRPR5 as the level of stability acquired by FATTA. Fig. 13
shows the calculated level of stability from the primary note to the present note. The
level of stability can be calculated after the third note because GTTM analysis needs
at least four notes.
Calculation Level of Stability of Melodies by FATTA. The predicted notes are
displayed in piano roll format within the range of view of the keyboard. The roll
scrolls down at a constant speed. Below the piano lid, which is made ofsemitransparent acrylic resin, there is a 32 × 25 full-color LED matrix for displaying
the scrolling piano roll. The 32 represents two measures when the resolution is a
sixteenth note, and 25 is the number of keys on the keyboard. The color of each LED
in the matrix is determined under the assumption that the onset of the next note will
start at the corresponding position on the piano roll and by calculating the level ofstability. When the level of stability is high, the LEDs show yellow, when it is low,
they show black, and when it is neither, they show red. There is also a 32 × 20 blue
LED matrix that displays the bar lines of the piano roll.
Construction. The piano is 953 mm long and 710 mm wide and resembles a grandpiano. It contains a MIDI keyboard, the LED display, a microcomputer, a power
supply, and four speakers. The LED display is 630 mm long and 390 mm wide. The
colors of the LEDs are controlled by MAX6972 which is 16-output 12-bit pulsewidth-modulation (PWM) LED drivers. There is a 5-mm gap between the LEDs andpiano lid to protect the acrylic resin lid from the heat of the LEDs. A half-mirror sheet
is bonded to the back side of the acrylic resin so that the lights of the LEDs show on
the surface of the piano lid rather than 5 mm below it. The LED drivers are controlledusing a microcomputer connected to the computer with a network cable. The
computer sends the data for the LED colors by using the user datagram protocol.

5

Melody Morphing

Initial melody A, target nuance melody B, morphing result melody C , and themorphing method must meet the following conditions. 1 and 2 are conditions for
melody C, 3 and 4 are conditions for the method.
1) The similarity between A and C is closer than that of A and B and thesimilarity between B and C is closer than that of A and B.
2) When B is the same as A, C will be the same as A.
3) The output of multiple melodies C depend on the parameters that decide
the level of influence of the features of melodies A and B.
4) C will be a monophony if A and B are monophonies.
5.1

Using GTTM for Melody Morphing

Our melody morphing method uses time-span trees acquired by analysing the resultsof the GTTM. Fig. 14 is an example of abstracting a melody by using a time-span tree.
In the figure, there is a time-span tree from melody D, which embodies the results of
the GTTM analyses. In the time-span tree, the important notes are connected to abranch nearer the root of the tree. In contrast, the un-important notes are connected to
the leaves of the tree. We can obtain an abstracted melody E by slicing the tree in the
middle and omitting notes that are connected to branches under line E. In the samemanner, if we slice the tree higher up at line F, we can get a more abstracted melody F.
We can regard the abstraction of melody as a kind of melody morphing, because
melody E is an intermediate melody between melody E and melody F.

Fig. 14. Abstraction of melody.

In order to realize the melody morphing, we use the primitive operations of thesubsumption relation (written as ⊑), meet (written as ⊓) and join (written as ⊔), as
proposed in Hirata [13]. The subsumption relation represents the relation "aninstantiated object" ⊑ "an abstract object" (Figure 15a). For example, the relationshipamong TD, TE and TF, which are the time-span trees (or reducted time-span trees) of
melodies D, E and F in Figure 2, can be represented as follows:
TF

⊑TE

⊑TD

(4)

The meet operator extracts the largest common part or the most common informationof the time-span trees of two melodies in a top-down manner (Figure 15b). The join
operator joins two time-span trees in the top-down manner as long as the structures of
two time-span trees are consistent (Figure 15c).
(a)

TA

(b)

TB

TA

(c) TB⊑
Instantiating Abstracting

TA ⊓ TBTA ⊔ TB

Fig. 15. Examples of subsumption ⊑ meet ⊓ and join ⊔ .

5.2

Overview of Melody Morphing Method

The meaning of morphing is to change one image into another through a seamless
transition. For example, a morphing method for a face picture can create intermediate
pictures through the following operations.
1) Link characteristic points such as on the eyes, nose, etc, in the two pictures(Fig. 16a).
2) Rate the intensities of shape (position), colour, etc, in each picture.
3) Combine the pictures.
Similarly, our melody morphing method creates intermediate melodies with thefollowing operations.
1) Link the most common information of the time-span trees of two melodies(Fig. 16b).
2) Abstract the notes of a melody in the difference branch of the time-span treeby using the melody divisional reduction method.
3) Combine both melodies.
The melody morphing method is illustrated in Fig. 17.

(b)

(a)

Fig. 16. Examples of linking two pictures/melodies.

TA

TB
TA

Linking common information of the melodies

TB
Melody division reduction

TC

TD

T
T
Melody D

Melody C
Combining two melodies

TA
TC TD
Melody B

Melody A

A broken line in a time-spantree indicates the branch has a
special operator [n1, n2].

Melody E

TB
Fig. 17. Overview of melody morphing method.

5.3

Linking Common Information of the Melodies

By using the time-span trees TA and TB from melodies A and B, we can calculate themost common information TA ⊓ TB which is not only the essential parts of melody A
but also those of melody B. The meet operation TA ⊓ TB are abstracted from TA and TB,
and those discarded notes are regarded to be the difference information of TA and TB.
We use FATTA [8] to generate a time-span tree from a score automatically. We
restrict the music structure to monophony, because FATTA only allows monophony
input. From now on, we use a word 'melody' only as it pertains to monophony.
When calculating TA ⊓ TB by extracting the largest common part of TA and TB in a
top-down manner, the result may change depending on whether or not the octave
notes such as C4 and C3 can be distinguished. If we discriminate octave notes, C4⊓ C3 will be empty ⊥ . On the other hand, if we do not discriminate octave notes, theresult is just C, which abstracts the octave information. We regard a note and theoctave note to be different notes, because processing is difficult if the octave
information is not defined.
5.4

Melody Division Reduction

We consider that there are features without the other melody in the difference
information of TA and TB. Therefore, we need a method for smoothly increasing or
decreasing the features. The melody divisional reduction method abstracts the notesof the melody in the difference branch of the time-span tree by applying theabstraction described in Section 5.1.
In the melody divisional reduction method, we can acquire melodies Cm(m=1,2,…,n) from TA and TA ⊓ TB, by following algorithm. The subscript m of Cm
indicates the number of notes in the difference information of the time-span trees thatare included in TCm and not included in TA ⊓ TB.
Step 1: Decide the level of abstraction
A user decides the parameter L that determines the level of abstraction of the melody.
L is from 1 to the number of notes in the difference information of the time-span treesthat are included in TA but not included in TA ⊓ TB.
Step 2: Abstraction of notes in the difference information
This step selects and abstracts a note which has the fewest number of dots in the
difference information. The numbers of dots can be acquired from the GTTM
analysis results [3]. If two or more notes have the fewest dots, we select the first one.
Step 3: Iteration
Iterate step 2 L times.
Subsumption relations hold as follows for the time-span trees TCm constructed withthe above algorithm.

TA

TB

TCn

TCn−1 … TC 2

TC1

TA

(5)

In Fig. 17, there are 9 notes included in TA but not included in TA ⊓ TB. Therefore,
the value of n is 8, and we can acquire eight kinds of melodies Cm (m=1,2,…,n)between TA and TA ⊓ TB. Hence, melody Cm is attenuates features that only have
melody A without melody B (Fig. 18).
In the same way, we can acquire melody D from TB and TA ⊓ TB as follows.

TA

5.5

TB

TD

TB

(6)

Combine Two Melodies

We use the join operator to combine melodies C and D, which are results of thedivisional reduction using time-span tree of melodies A and B. The simple join
operator is not sufficient for combining TC and TD, because TC ⊔ TD is not always amonophony if TC and TD are monophonies. In other words, the result of the operation
has chords when the time-span structures are overrides and the pitches of the notes aredifferent; therefore, the result violates condition 4 in section 3.
In order to solve this problem, we introduce a special operator [n1, n2], whichindicates note n1 or note n2, as a result of n1 ⊔ n2. Then, the result of TC ⊔ TD is all
combinations of monophonies made by the operators.
5.5

ShakeGuitar

The ShakeGuitar is demonstration system of melody morphing method for use with
iPhone/iTouch that enables users to enjoy the simulated experience of guitar playing,even if they are musical novices (Fig. 19). Shaking the iPhone/iTouch with varying
degrees of strength influences the level of guitar-melody difficulty, which smoothly
changes in real-time from soft backing to heavy soloing by using melody morphing
method. Users can identify the difficulty level visually as the color of the guitar bodychanges corresponding to the level.

TA

LA＝1

LA＝2

LA＝3

LA＝4

LA＝5

LA＝6

LA＝7

LA＝8

TA

TB

Fig. 19. Melody divisional reduction.

Fig. 19. ShakeGuitar.

6

Experimental Results

It is difficult to compare the performance of our system with those of previous
systems, because the approaches taken are completely different. Our method, which isbased on music theory, evaluates the appropriateness of the notes from a musicalpoint of view. Therefore, we first quantitatively evaluate each step in our method andthen describe an example result.
6.1

Evaluation of ATTA and FATTA

We evaluated the performance of the music analyzer using an F-measure, which is
given by the weighted harmonic mean of Precision and Recall,

Fmeasure = 2 ×

P× R
P+R

(6)

This evaluation required us to prepare correct data of a grouping structure, metricalstructure, and time-span tree. We collected a hundred pieces of 8-bar length,monophonic, classical music pieces, and asked musicology experts to manually analyze
them with faithful regard to the GTTM, using the manual-edit mode of interactive
GTTM analyzer to assist in editing the grouping structure, metrical structure, and timespan tree. Three other experts crosschecked these manually produced results.
To evaluate the baseline performance of our system, we used the following default
parameters: Srules=0.5, Trules=0.5, Ws,=0.5 Wr =0.5, Wl=0.5, and σ=0.05.
In the current stage, the parameters are configured by humans, because the optimalvalues of the parameters depend on a piece of music. When a user changes theparameters, the hierarchical structures change as a result of the new analysis.
It took us an average of about 10 minutes per piece to find the plausible tuning forthe set of parameters (Table 1,2 and 3). As a result of configuring the parameters,
each F-measure of our analyzer outperformed the baseline (Table 4).
Table 4. F-measure of our analyzer outperformed the baseline.
Melodies
1. Moments musicaux
2. Wiegenlied
3. Traumerei
4. An die Freude
5. Barcarolle

Total (100 melodies)Grouping Structure Analyzer

Metrical Structure Analyzer

Time-Span Tree Analyzer

Baseline
performance

Baseline
performance

Baseline
performance

Our method
with
Configured
parameters

0.71
0.54
0.50
0.22
0.24
:
:
0.44

0.84
0.69
0.63
0.48
0.60
:
:
0.60

0.18
0.76
0.60
0.12
0.04
:
:
0.46

Our method
with
Configured
parameters

0.56
1.00
0.87
0.73
0.54
:
:
0.77

0.95
0.83
0.76
0.95
0.72
:
:
0.84

Our method
with
Configured
parameters

1.00
0.85
1.00
1.00
0.79
:
:
0.90

After the set of parameters was optimized using FATTA, the average F-measure was
0.48, 0.89, and 0.49, respectively, for grouping, metrical, and time-span tree
structures, all better than the baseline performance of ATTA.

6.2

Evaluation of Interactive GTTM Analyzer

We measured the operating time for acquiring the target analysis results of ourinteractive GTTM analyzer and compared it with that of the GTTM manual editor
without an ATTA. For the target analysis, we used one hundred pieces from the threehundred pairs of scores and correct data of grouping structure, metrical structure, andtime-span tree. We did not use the prolongational tree in this measurement since its
analyzer is still under development. As a result, our interactive GTTM analyzer
outperformed the GTTM manual editor without an ATTA (Table 5).
Table 5. Operation time of interactive GTTM analyzer and GTTM manual editor.
Interactive
analyzer

Melodies
1 Grande Valse Brillante
2. Moments Musicaux
3. Turkish March
4. Anitras Tanz
5. Valse du Petit Chien
Total (100 melodies)

6.3

GTTM

GTTM
editor326 sec
541 sec
724 sec
621 sec
876sec.
:575 sec.

manual624 sec
791 sec.
1026 sec
915 sec.
1246 sec.
:
891 sec.

Example of Melody Expectation

Fig. 19 shows the results for Haydn’s andante. The graph below the musical staff
indicates the level of melody stability, corresponding each above note. The number
under each note indicates the level of stability for selecting a pitch of 25 possible
ones. Stability of a pitch event is calculated from the partial melody between its
beginning and the relevant event, and not by the sole event. As a result, the F# inmeasure 7 does not have the lowest stability, and that of C in measure 5 is lower
than that of G in the previous measure, even in C major. Although this may
contradict intuition, the calculated result is faithful to our algorithm. At the end of
the 4th and 8th measures, the level of stability is high. This is because a dominantchord’s note that wants resolve to a tonic chord’s note occurs. In contrast, at the
beginning of the 5th measure, the level of stability is relatively low. This is because
a tonic chord’s root note at the beginning of the 5th measure occurs, and various
progressions can follow the root note. These results show that our prediction
method works well from a musical point of view.

5 6 4 3

Level of stability

5 4 5 3 4 3 2

4 8 7 44 2 4 8

7

5 2

1.0
0

Time
Fig. 19. Example of melody expectation.

6.1

Evaluation of Melody Morphing Method

We tried to determine whether the method can generate the extrapolative melody Mfrom melody A and B which holds following expression.

{R( A, M ) < R( A, B) and R( B, M ) < R( A, B)}(7)

where R(X,Y) indicates the similarity between melodies X and Y. The meaning of theexpression (7) is that a melody M is an interpolative melody of melodies A and B.
To measure the similarity between melodies X and Y, we used the following RN(X,Y), defined by Hirata [13], which indicates how much information is lacking from thetwo melodies as a result of the meet operation.

RN ( X , Y ) =meet ( X , Y ) N(

m a x X N,Y N)(8)

where |X|N indicates the number of notes in melody X.
We use 10 pairs of sample melodies A and B, and as a result of confirmation, all
the extrapolative melodies M from melodies A and B, holds the expression (7).

7Conclusion

We developed a music analyzing system called ATTA and FATTA, which derives thetime-span tree of GTTM. We constructed a music analyzer called the interactive
GTTM analyzer, which derives the grouping structure, metrical structure, time-spantree, and prolongational tree of the GTTM. The analyzer also derives analysis resultsof chord progression based on the Tonal Pitch Space. We also proposed melody
expectation method and melody morphing method, which uses the time-span tree of
GTTM.
The following five points are the main results of this study.
Proposed extended GTTM. We propose an extended GTTM for computer implementation. The difficulty with computer imple-mentation of GTTM has been
designated, however no radical solution has been proposed. We re-formalized therules using a numerical expression with adjustable parameters, so that it can separate
the definition and ambiguity from the material analyzed.
Implemented Music Analyzer on computer. We implemented an actual working system
to ac-quire the hierarchical grouping structure, metrical structure, and time-span tree. The
iteractive GTTM analyzer consists of an automatic GTTM analyzer, called an ATTA, a GTTM
manual editor, and a GTTM process editor. By using the process editor, a user can seamlessly
change the analysis process of the ATTA and that of the manual editor. The experimental

results show that our interactive GTTM analyzer outperformed the GTTM manual editor
without an ATTA.

Experimented with correct data. Our experimental results showed that, as a resultof configuring the parameters, our music analyzer out-performed the baseline Fmeasure. We made a set of one hundred correct data, which is the greatest database ofanalyzed results of GTTM thus far. We plan to exhibit the database with the analyzer
in the near future. Since we hope to contribute to the research of music analysis, wepublicize our interactive GTTM analyzer and a dataset of three hundred pairs of a
score
and
analysis
results
bymusicologist
onourwebsitehttp://music.iit.tsukuba.ac.jp/hamanaka/gttm.htm, which is the largest database ofanalyzed results from the GTTM to date.
Proposed the Melody Expectation Method. We devised a melody expectation
method that predicts candidate notes on the basis of the GTTM and the tonal pitch
space (TPS). It is designed to be used with an expectation piano, which displays thepredicted notes on its lid, thereby supporting musical novices in playing
improvisations.
Proposed the Melody Morphing Method. We constructed a melody morphing
method for generating interpolative melodies from two input melodies A and B byusing melody divisional reduction, which smoothly decreases the difference
information of those melodies. To generate, interpolative melodies, all we need to do
is select two input melodies and configure the parameters for controlling the reduction
or augmentation level of each melody. We plan to finish developing the interactive
melody generator and evaluate whether the melody-morphing method is useful for
generating melodies.

We plan to develop further systems, using time-span trees and the results of the music
analyzer, for other musical tasks such as harmonizing, voicing, ad-lib, and so on, to
indicate the effectiveness of implementing the GTTM to provide music knowledge.

References
1. Apple - GarageBand, http://www.apple.com/ilife/garageband/.
2. Todd, N. : A Model of Expressive Timing in Tonal Music, Musical Perception, Vol. 3, No.
1, pp. 33--58 (1985).
3. Widmer, G.: Understanding and Learning Musical Expression, In Proceeding of the 1983
International Computer Music Conference (ICMC1983), pp. 268--275, New York (1983).
4. Hirata, K. and Hiraga, R: Ha-Hi-Hun plays Chopin's Etude, In Working Notes of IJCAI-03
Workshop on Methods for Automatic Music Performance and their Applications in a Public
Rendering Contest, pp. 72--73, Acapulco (2003).

5. Hirata, K. and Matsuda, S.: Interactive Music Summarization based on Generative Theory ofTonal Music, Journal of New Music Research (JNMR), Vol. 32, No. 2, pp. 165--177 (2003).
6. Hamanaka, M., Hirata, K., and Tojo, T.: Implementing "A Generating Theory of Tonal
Music", Jornal of New Music Reserch (JNMR), Vol. 35, No. 4, pp. 249--277 (2007).
7. Hamanaka, M., Hirata, K., and Tojo, S.: ATTA: Automatic Time-span Tree Analyzer basedon Extended GTTM, In Proceedings of the 6th International Conference on Music
Information Retrieval Conference (ISMIR2005), pp. 358--365, London (2005).
8. Hamanaka, M., Hirata, K., and Tojo, S.: FATTA: Full Automatic Time-span Tree Analyzer,
Proceedings of the 2007 International Computer Music conference (ICMC2007), Vol. 1,
pp. 153--156, Copenhagen (2007).
9. Lerdahl, F., and Jackendoff, R. A Generative Theory of Tonal Music. Cambridge,
Massachusetts: MIT Press (1983).
10. Balaban, M.: The Music Structures Approach to Knowledge Representation for Music
Processing', Computer Music Journal, 30:2, pp. 96--111 (1996).
11. Cope, D.: Experiments in Musical Intelligence. A-R Editions, Inc. (1996).
12. Dannenberg, R.: Machine Tongues XIX: Nyquist, a Language for Composition and Sound
Synthesis', Computer Music Journal, 21:3, pp. 50--60 (1997).
13. Hirata, K., and Aoyagi, T.: Computational Music Representation Based on the Generative
Theory of Tonal Music and the Deductive Object-Oriented Database, Computer Music
Journal, 27:3, pp. 73--89 (2003).
14. Hamanaka, M., Hirata, K., and Tojo, S.: Melody Extrapolation in GTTM Approach,
Proceedings of the 2009 International Computer Music Conference (ICMC2009), pp. 89--92,
Montreal (2009).
15. Hamanaka, M., Hirata, K., and Tojo, S.: Melody Morphing Method based on GTTM,
Proceedings of the 2008 International Computer Music conference (ICMC2008), pp. 155-158, Belfast (2008).
16. Mozer, M.: Neural Network Music Composition by Prediction: Exploring the Benefits of
Psychoacoustic Constraints and Multi-Scale Processing, Connection-Science, Vol. 6, No. 23, pp. 247--280 (1994).
17. Yen, G., Lucas, S., Fogel, G., Kendall, G., Salomon, R., Zhang, B., Coello, C., andRunarsson, T.: Evolving Musical Sequences with N-Gram Based Trainable Fitness
Functions, Proceedings of the 2006 IEEE Congress on Evolutionary Computation, pp. 604-614 (2006).
18. Cooper, G., and Meyer, L. B.: The Rhythmic Structure of Music, The University of
Chicago Press (1960).
19. Narmour, E.: The Analysis and Cognition of Basic Melodic Structure. The University of
Chicago Press (1990).
20. Temperley, D.: The Cognition of Basic Musical Structures. MIT press, Cambridge (2001).
21. Larson, S.: Musical Forces and Melodic Expectations: Comparing Computer Models with
Experimental Results, Music Perception, 21/4, pp. 457--498 (2004).
22. Lerdahl, F.: Tonal Pitch Space, Oxford University Press (2001).
23. Hamanaka, M., Hirata, K. and Tojo, S.: Automatic Generation of Grouping Structure basedon the GTTM, In Proceeding of 2004 International Computer Music Conference(ICMC2004) pp. 141--144 (2004).
24. Hamanaka, M., Hirata, K. and Tojo, S.: Automatic Generation of Metrical Structure basedon the GTTM, In Proceeding of 2005 International Computer Music Conference(ICMC2005), pp. 53--56 (2005).
25. Sakamoto, S., and Tojo, S.: Harmony Analysis of Music in Tonal Pitch Space, Information
Processing Society of Japan SIG Technical Report, Vol. 2009 (2009) (in Japanese).
26. Hamanaka, M., Goto, M., Asoh, H., and Otsu, N.: 'A Learning-Based Quantization:
Unsupervised Estimation of the Model Parameters, In Proceedings of 2003 International
Computer Music Conference (ICMC2003), pp. 369--372, Singapore (2003).