MELODY MORPHING METHOD BASED ON GTTM
Masatoshi Hamanaka
University of Tsukuba
hamanaka@iit.tsukuba.ac.jp

Keiji Hirata
NTT Communication ScienceLaboratories

ABSTRACTThis paper describes a melody morphing method thatgenerates an intermediate melody between a melody andanother melody with a systematic order according to acertain numerical measure. Conventional music sequence
software only operates on the surface structure of music,
such as its notes and rests. The time-span tree, which isacquired from the music surface by using a music theory
called GTTM (Generative Theory of Tonal Music),enables us to analyze the deeper structure. Our method
makes it possible for one melody to be morphed intoanother melody by using a melody divisional reductionthat applies meet and join operations to the GTTM timespan trees. Experimental results show that our morphing
method makes it possible to generate intermediate
melodies between two melodies.
1.

INTRODUCTION

Our goal is to create a system that will enable a musicalnovice to manipulate a piece of music, which is an
ambiguous and subjective media, according to his or her
intentions. We believe that it is important to make it possible
to create a musical system for musical novices that can:1) easily manipulate a piece of music, and2) mirror the user's intentions.
Note that the higher the abstraction level of the objects
for manipulating the music is, the more difficult itbecomes to reflect the user's intentions. For example, it
is difficult for musical novices to manipulate music withcommercial music sequence software that only operates
on the surface structure of music, that is, the pitch andnote-on timing of each note. On the other hand,
Garageband [1] can create a piece of music thoughsimple manipulations, i.e., by just concatenating prestored phrases. However, when we want arrange aportion of a melody in a phrase, we have to manipulate
the surface structure of the music, and a musical novice
would find it difficult for the software to mirror his or
her intentions in such a case.
We constructed a system that will enable a musical novice
to manipulate a piece of music by using a music theory
called the Generative Theory of Tonal Music (GTTM) [2].
We have previously developed a system of music analysis
based on GTTM called FATTA [3-4]. FATTA cangenerate a time-span tree as the result of a GTTM analysis.
GTTM consistently represents multiple aspects of musicin a single framework. This feature is important whenthe musical system is to assist a musical novice inmanipulating musical structures. For instance, if we
imagine a simple operation that splits a melody, the split
operations may vary depending on the relevant musicalstructure. Therefore, it is preferred that the splitting

Satoshi Tojo
Japan Advanced Institute ofScience and Technology

position of the melody and that of the ornamented melody
be identical. Unless a system of consistent operations interms of melody, rhythm, and harmony is developed, theresulting splitting position may be different.
With regard to how the software should reflect the user's
intentions, we take an example of composing a melody,
where it is supposed that a user wants to arrange melody
A by adding some musical nuances to it and he/she knows
melody B has such nuances. If a user could use a system
accepting a morphing command, issuing a simple
command, for example, "add the nuance of melody B tomelody A", can accurately convey a user's intention to amorphing system. As a result, the system generates
multiple melodies approaching from melody A to B bylittle and little. The advantage of morphing is not only its
simple and accurate transferability and manipulability but
also its ease of understanding the relationship between
inputs and outputs. Our system exploits melody morphing
to make it possible for novices to create melodies
reflecting their intensions.
Previous music systems [5-7] have their own way ofmusic analysis, from which it is difficult to acquire
deeper musical structures, and thus these systems aredifficult to manipulate according to the user's intention.
On the other hand, Hirata [8] defined a representation
method and primitive operation for polyphony, and thisdevelopment indicates the potential for constructing a
melody arranging algorithm.
In this paper we developed a melody morphing method
in which we input monophonies A and B and thengenerate intermediate melodies between melodies A andB with a systematic order according to a certain
numerical measure by configuring the parameters that
determine the level of influence of the features ofmelodies A and B. As part of this overall method, we
devised the melody divisional reduction method toreduce the notes of melody A in the difference branch of
the time-span tree of melodies A and B.
2.

GTTM

Melody morphing uses time-span trees acquired byanalysing the results of the Generative Theory of Tonal
Music (GTTM). In this section, we briefly describe
GTTM. GTTM is composed of four modules, each ofwhich assigns a separate structural description to alistener's understanding of a piece of music. These four
modules output a grouping structure, a metrical structure,a time-span tree, and a prolongational tree, respectively(Figure 1). The time-span tree is a binary tree, which is ahierarchical structure describing the relative structural
importance of notes that differentiate the essential partsof the melody from the ornamentation.

Time-span tree…
・・ ・ ・・ ・ ・・ ・ ・・ ・ ・・ ・ ・・ ・ ・・ ・ ・・ ・・・・・・・・

Metrical structureGrouping structure

Figure 1. Time-span tree, metrical structure, andgrouping structure.

2.1. Abstraction of melody
Figure 2 is an example of abstracting a melody by using atime-span tree. In the figure, there is a time-span tree frommelody D, which embodies the results of the GTTM
analyses. In the time-span tree, the important notes areconnected to a branch nearer the root of the tree. In contrast,
the un-important notes are connected to the leaves of thetree. We can obtain an abstracted melody E by slicing thetree in the middle and omitting notes that are connected tobranches under line E. In the same manner, if we slice thetree higher up at line F, we can get a more abstracted
melody F. We can regard the abstraction of melody as akind of melody morphing, because melody E is anintermediate melody between melody E and melody F.

Figure 2. Abstraction of melody.

2.2. Primitive operations using time-span trees
In order to realize the melody morphing, we use theprimitive operations of the subsumption relation (writtenas ⊑ ), meet (written as ⊓ ) and join (written as ⊔ ), as
proposed in Hirata [8]. The subsumption relation
represents the relation "an instantiated object" ⊑ "anabstract object" (Figure 3a). For example, the relationshipamong TD, TE and TF, which are the time-span trees (orreducted time-span trees) of melodies D, E and F in
Figure 2, can be represented as follows:
TF ⊑ TE ⊑ TD
The meet operator extracts the largest common part or themost common information of the time-span trees of two
melodies in a top-down manner (Figure 3b). The join operator
joins two time-span trees in the top-down manner as long as
the structures of two time-span trees are consistent (Figure 3c).
(a)

TA(b)

TB

TA

(c)

TB

⊑
Instantiating Abstracting

TA ⊓ TBTA ⊔ TB

Figure 3. Examples of subsumption ⊑ ,meet ⊓ and join ⊔ .

3.

MORPHING METHOD BASED ON GTTM

Initial melody A, target nuance melody B, morphing
result melody C , and the morphing method must meetthe following conditions. 1 and 2 are conditions for
melody C, 3 and 4 are conditions for the method.
1. The similarity between A and C is closer than that ofA and B and the similarity between B and C is closer
than that of A and B.
2. When B is the same as A, C will be the same as A.
3. The output of multiple melodies C depend on theparameters that decide the level of influence of thefeatures of melodies A and B.
4. C will be a monophony if A and B are monophonies.
3.1. Overview of melody morphing method
The meaning of morphing is to change one image intoanother through a seamless transition. For example, amorphing method for a face picture can create
intermediate pictures through the following operations.
1) Link characteristic points such as on the eyes, nose,etc, in the two pictures (Figure 4a).
2) Rate the intensities of shape (position), colour, etc, ineach picture.
3) Combine the pictures.
Similarly, our melody morphing method createsintermediate melodies with the following operations.
1) Link the most common information of the time-spantrees of two melodies (Figure 4b).
2) Abstract the notes of a melody in the difference
branch of the time-span tree by using the melody
divisional reduction method.
3) Combine both melodies.
The melody morphing method is illustrated in Figure 5.
(a)

(b)

Figure 4. Examples of linking two pictures / melodies.

3.2. Linking common information of the melodies
By using the time-span trees TA and TB from melodies A andB, we can calculate the most common information TA ⊓ TB
which is not only the essential parts of melody A but also
those of melody B. The meet operation TA ⊓ TB areabstracted from TA and TB, and those discarded notes areregarded to be the difference information of TA and TB.
We use FATTA [4] to generate a time-span tree from ascore automatically. We restrict the music structure tomonophony, because FATTA only allows monophony
input. From now on, we use a word 'melody' only as it
pertains to monophony.

TA TBTA TB

3.2 Linking common information of the melodies

TDTCTC

3.3 Melody division reduction

TD
Melody D

Melody C
3.4 Combining two melodies

TATB
TC TD
Melody B

Melody A

A broken line in a time-span tree indicatesthe branch has a special operator [n1, n2].

Melody EFigure 5. Overview of melody morphing method.
When calculating TA ⊓ TB by extracting the largest
common part of TA and TB in a top-down manner, theresult may change depending on whether or not theoctave notes such as C4 and C3 can be distinguished. If
we discriminate octave notes, C4 ⊓ C3 will be empty ⊥ .
On the other hand, if we do not discriminate octave
notes, the result is just C, which abstracts the octave
information. We regard a note and the octave note to bedifferent notes, because processing is difficult if theoctave information is not defined.
3.3. Melody division reduction
We consider that there are features without the othermelody in the difference information of TA and TB.
Therefore, we need a method for smoothly increasing ordecreasing the features. The melody divisional reduction
method abstracts the notes of the melody in thedifference branch of the time-span tree by applying theabstraction described in Section 2.1.
In the melody divisional reduction method, we canacquire melodies Cm (m=1,2,…,n) from TA and TA ⊓ TB,
by following algorithm. The subscript m of Cm indicatesthe number of notes in the difference information of thetime-span trees that are included in TCm and not includedin TA ⊓ TB.
Step 1: Decide the level of abstraction
A user decides the parameter L that determines the levelof abstraction of the melody. L is from 1 to the numberof notes in the difference information of the time-spantrees that are included in TA but not included in TA ⊓ TB.

Step 2: Abstraction of notes in the difference information
This step selects and abstracts a note which has the fewest
number of dots in the difference information. The numbers ofdots can be acquired from the GTTM analysis results [3]. Iftwo or more notes have the fewest dots, we select the first one.
Step 3: Iteration
Iterate step 2 L times.
Subsumption relations hold as follows for the time-spantrees TCm constructed with the above algorithm.

TATB

TCn

TCn−1 … TC 2

TC1

TA

In Figure 5, there are 9 notes included in TA but not includedin TA ⊓ TB. Therefore, the value of n is 8, and we can acquire
eight kinds of melodies Cm (m=1,2,…,n) between TA and TA⊓ TB. Hence, melody Cm is attenuates features that only have
melody A without melody B (Figure 6).
In the same way, we can acquire melody D from TB andTA ⊓ TB as follows.

TATBTDTB

3.4. Combining two melodies
We use the join operator to combine melodies C and D,
which are results of the divisional reduction using time-spantree of melodies A and B. The simple join operator is not
sufficient for combining TC and TD, because TC ⊔ TD is notalways a monophony if TC and TD are monophonies. In otherwords, the result of the operation has chords when the timespan structures are overrides and the pitches of the notes aredifferent; therefore, the result violates condition 4 in section 3.

In order to solve this problem, we introduce a specialoperator [n1, n2], which indicates note n1 or note n2, asa result of n1 ⊔ n2. Then, the result of TC ⊔ TD is all
combinations of monophonies made by the operators.

TALA＝1

4.

EXPERIMENTAL RESULTS

We tried to determine whether melody C, which is themorphing result of melodies A and B, is theirintermediate. Namely, we tried to determine whether ourmelody morphing method fulfils condition 1 at thebeginning of section 3. To measure the similarity
between A and B, A and C, and B and C, we used thefollowing RN(A,B), defined by Hirata [8], whichindicates how much information is lacking from the two
melodies as a result of the meet operation.
meet ( A, B) N(1)R ( A, B ) =N(ma x AN, B N)

where |A|N indicates the number of notes in melody A.
We use 10 pairs of sample melodies A and B, and as aresult of confirmation, our morphing method meets
condition 1 in Section 3 for any parameter set of LA and LB.

LA＝2

5.

CONCLUSION

We devised a melody morphing method based on GTTM,
which makes it possible to construct an intermediate
melody between one melody and another melody.
Experiment results show that the morphed melodies areintermediates of the input two melodies according to thesimilarity defined by Hirata [8]. In fact, we do not thinkthat current experimental results are not enough to support
our claim that the morphing method proposed meets theconditions listed in Section 3 under the new metricdefined in Section 4. Since we however believe that ourmethod has a potential to support the claim, we would liketo conduct another experiment to show justification. We
are now planning to extend the method to polyphony,
because we have thus far restricted it to monophony.

LA＝3

LA＝4

LA＝5

6.
LA＝6

LA＝7

LA＝8

TA TB

Figure 6. Melody divisional reduction.

REFERENCES[1] http://www.apple.com/ilife/garageband/.
[2] Lerdahl, F., and Jackendoff, R. A Generative Theory ofTonal Music. Cambridge, Massachusetts: MIT Press, 1983.
[3] Hamanaka, M., Hirata, K., and Tojo, S. ''Implementing‘A Generative Theory of Tonal Music’'', Journal of New
Music Research, 35:4, 249-277, 2006.
[4] Hamanaka, M., Hirata, K., and Tojo, S. ''FATTA:
Full Automatic Time-span Tree Analyzer'',
Proceedings of the International Computer Music
Conference, Vol. 1, pp. 153-156, 2007.
[5] Balaban, M. '' The Music Structures Approach to
Knowledge Representation for Music Processing'',
Computer Music Journal, 30:2, pp. 96-111, 1996.
[6] Cope, D. Experiments in Musical Intelligence. A-R
Editions, Inc. 1996.
[7] Dannenberg, R. '' Machine Tongues XIX: Nyquist, aLanguage for Composition and Sound Synthesis'',Computer Music Journal, 21:3, pp. 50-60, 1997.
[8] Hirata, K., and Aoyagi, T. '' Computational Music
Representation Based on the Generative Theory of Tonal
Music and the Deductive Object-Oriented Database'',Computer Music Journal, 27:3, pp. 73-89, 2003.