Melody Slot Machine:
Melody Morphing by Using Time-span Tree of GTTM
Masatoshi Hamanaka
RIKEN

Mizuki Kobayashi
Kyoko Otagawa
Mayumi Shimada
RIKEN/Toho Gakuen ColRIKEN
RIKEN
lege Music Department
masatoshi.hamanaka@riken.jp
fore, for 15 years, we have been implementing the GTTM,
ABSTRACT
proposed by Fred Lerdahl and Ray Jackendoff in 1983 [4This paper describes an overview of our developed inter7]. The performance of GTTM analysis by computers has
active music system called a "Melody Slot Machine,"
recently been dramatically improved by introducing deep
which is an application based on the generative theory oflearning [7].
tonal music (GTTM). Generally, melody manipulation
The time-span tree, which is the deep structure obtained
requires expert knowledge and is difficult for musicalas a result of GTTM analysis, shows the relationship benovices. Therefore, we developed the Melody Slot Matween the main notes and the ornamatation notes strucchine, which provides an experience of manipulatingturally. The main advantage of using the time-span tree is
melodies by allowing users to freely switch two originalthat it is possible to reduce the notes of a melody by demelodies and morphing melodies. We conducted an excreasing the ornamentation notes step by step while mainperiment with a musicologist who understood the melody
taining the main structure of the melody. One example ofmorphing method based on GTTM and attempted to use ita melody operation by using the time-span tree is melody
for composition. When a melody generated with themorphing, which generates an intermediate melody frommethod was unnatural, the musicologist added notes to
two input melodies. Previously, we proposed a melody
correct it. The experimental results showed that 78.5% ofmorphing method to generate an intermediate melody ofthe notes in the composed pieces were generated with thetwo melodies as a melody operation that uses the timemelody morphing method, while the remaining 21.5%
span tree [8]. We are also studying an operation called
was added by the musicologist.
flip to invert the time-span tree [9]. We are planning torealize the operation of various melodies in the future by
1. INTRODUCTION
using time-span trees.
We believe that melody manipulation by the time-span
This paper describes a system called the "Melody Slottrees can give pleasure to musical novices by enabling
Machine," which enables novices without musicalthem to manipulate music and can improve the composiknowledge to manipulate melodies. In the Melody Slottion efficiency of professional composers. To assess
Machine, melody fragments are displayed on the dial, andwhether the former is feasible, we construct a system
the melody to be played can be switched by rotating thecalled a Melody Slot Machine. We conducted an experidial. The valuations of melody fragments are composed
ment to verify the effectiveness of the melody morphingon the basis of the generative theory of tonal music
method. In the experiment, a musicologist who under(GTTM) [1], and thus, switching the melody fragments
stood the method used it to manually generate an interdoes not change the overall structure of the melody.
mediate melody. If the intermediate melody was unnatuVarious melody generation systems based on deepral, the musicologist added notes to correct it. We foundlearning are being constructed [2, 3]. For example,
that 78.5% of the notes were generated with the melody
DeepBach enables users to generate three voices frommorphing method, while the remaining 21.5% was manuone voice by merging the learned networks of two longally added by the musicologist.
short-terms memories (LSTMs) and one neural network[2]. In another example, Celtic Melody Generation makes
2. MELODY SLOT MACHINE
it possible to generate melodies of Celtic-style ethnic
music using a recurrent neural network (RNN) [3]. AltThe Melody Slot Machine, which enables users to mahough those systems performed well in specific tasks,
nipulate melodies, has three features.
they are not compatible with various music operations
and are difficult at this stage for people to use to compose
2.1 User friendly interfaces
music [2, 3].
To allow anyone to easily manipulate melodies, we
We believe that a computer that understands music
adopted a dial type interface that makes it possible todeeply must be developed to realize a composition supreplace a part of the melody (Figure 1). There is a rectanport system that enables various music operations. Theregular hole in a part of the acrylic board sandwiching the
Copyright: © 2019 Masatoshi Hamanaka et al. This is an open-accessscore, and the dial interface on a tablet can be operated
article distributed under the terms of the Creative Commonswith fingers through the hole. When the red lever on the
Attribution License 3.0 Unported, which permits
right side of the score is pulled down, all the dials rotate,
unrestricted use, distribution, and reproduction in any medium, provided
and one of the melody fragments on the dial is randomly
the original author and source are credited.
selected.

3. TIME-SPAN TREE OF GTTM

Figure 1. Slot dial and lever

Melody morphing uses time-span trees obtained from theresults of GTTM analysis. The GTTM consists of four
modules, each of which assigns a separate structural description to a listener's understanding of a piece of music.
As shown in Figure 4, the four modules output a grouping structure, metrical structure, time-span tree, and prolongational tree. The time-span tree is a binary tree,
which is a hierarchical structure, representing the relative
structural importance of notes that differentiate the essential parts of a melody from ornamentation.

2.2 Easy-to-understand operation results
We prepare a display showing a performer so that theresult of the operation can be confirmed visually as well
as aurally. A holographic display was used for show theperformer so as to increase the feeling of presence (Figure 2). The users can feel like they are controlling a performer by operating a melody.

Prolongation tree

Time-span tree

Metrical structureGrouping structure

Figure 4. Prolongational tree, time-span tree, metricalstructure, and grouping structure obtained from GTTM
analysis

3.1 Abstraction of melody

Figure 2. Holographic display

2.3 Improve the feeling of presence
We recorded all the performance sounds so as to increase
the feeling of presence. For the recording, we used a studio with very little reverberation because otherwise only
the reverberation of the preceding sound enters the beginning of the melody fragment due to the melody splitting into fragments. Reverberation is added when the
melody is played. Three pairs of speakers were installed,
and panpot and reverb were set for each direction so thatthe holograph seemed to be a real performer (Figure 3).
When putting your head between two pairs of speakers,
both the sound and video enhance the presence.

Figure 5 shows an example of abstracting a melody byusing a time-span tree. The figure includes a time-spantree from melody, D, which embodies the results of
GTTM analysis. In the time-span tree, important notes
are connected to branches nearer the root of the tree,
whereas unimportant notes are connected to leaves. We
can obtain an abstracted melody, E, by slicing the tree inthe middle (line E) and then omitting notes whose branchconnections are below line E. In the same manner, if we
slice the tree higher up at line F, we can obtain an evenmore abstracted melody, F. We can regard this abstractionof melody as a kind of melody morphing because melody
E is an intermediate melody between melodies D and F.

Figure 5. Abstraction of melody

3.2 Primitive operations of time-span trees

Figure 3. Three pairs of speaker

To implement melody morphing, we use the primitive operations: the subsumption relation (written as ⊑), meet (written as ⊓), and join (written as ⊔ ) [9]. As shown in Figure 6a,subsumption represents the relation by which "an instantiated object subsumes an abstract object." For example, therelationship among TD, TE, and TF, which are the timespan trees (or reduced time-span trees) of melodies D, E, andF in Figure 5, can be represented as follows:
TF ⊑ TE ⊑ TD

(1)

Figure 6b illustrates the meet operator, which extracts thelargest common part or most common information of thetime-span trees of two melodies in a top-down manner.
Finally, Figure 6c illustrates the join operator, which
joins two time-span trees in a top-down manner as longas their structures are consistent.
(a)

(b)

TA

⊑
Instantiating Abstracting

TA ⊓ TB

(c)

TB

TA

TB

TA ⊔ TB

Figure 6. Examples of subsumption ⊑, meet ⊓, and join
⊔ operations

4. MELODY MORPHING METHODBASED ON GTTM

In this section, we explain the melody morphing method
based on GTTM we proposed in 2008 [8]. The initial
melody A, target nuance melody B, morphing result melody C, and melody morphing method must meet the following conditions: 1 and 2 for melody C, and 3 and 4 forthe method.
1. A must be more similar to C than to B, and B mustalso be more similar to C than to A.
2. When B is the same as A, C will be the same as A.
3. The output of multiple Cs depends on parametersthat determine the levels of influence of the featuresof A and B.
4. C will exhibit monophony if A and B are monophonic.
4.1 Overview of melody morphing method
Morphing means to change one image into anotherthrough a seamless transition. For example, a morphing
method for a facial image can create intermediate images
through the following operations.
1. Link characteristic points such as the eyes and nosein two images, as shown in Figure 7a.
2. Rate the intensities of shape (position), color, and soforth in each image.
3. Combine the images.
(b)(a)

Figure 7. Examples of linking two images andtwo melodies

Similarly, our melody morphing method creates intermediate melodies through the following operations.
1. Link the most common information of the time-spantrees of two melodies, as shown in Figure 7b.
2. Abstract the notes of a melody in a differing branchof the time-span tree by using the melody divisional
reduction step of our melody morphing method.
3. Combine both melodies.
Figure 8 illustrates the melody morphing method.
4.2 Linking of common information
By using the respective time-span trees TA and TB frommelodies A and B, we can calculate the most common
information TA ⊓ TB, which includes the essential parts of
not only A but also B. The meet operation TA ⊓ TB abstracts common notes from TA and TB, and the discarded
notes are then regarded as the difference information of
TA and TB.
When calculating TA ⊓ TB by extracting the largest
common part of TA and TB in a top-down manner, theresult may change depending on whether octave notes
such as C4 and C3 can be distinguished. If we discriminate octave notes, then C4 ⊓C3 will be empty, denoted as⊥. On the other hand, if we do not discriminate octave
notes, the result is just C, which abstracts the octave information. Here, we regard a note and its octave as different notes, because processing is difficult if the octave
information is not defined.
4.3 Melody divisional reduction
We next consider that the difference information of TA
and TB includes features not present in the other respective melody. Therefore, we need a method for smoothly
inserting or removing such features. The melody divisional reduction step of our melody morphing method
abstracts the notes of the melody in a difference branch of
the time-span tree by applying the abstraction describedin Section 3.1.
Using this method, we can acquire melodies Cm(m=1,2,…,n) from TA and TA ⊓TB with the following algorithm. The subscript m indicates the number of notes inthe difference information of the time-span trees that areincluded in TCm but not in TA ⊓TB.
Step 1: Determine the level of abstraction.
The user selects a parameter L that determines the levelof abstraction of the melody. L can range from 1 to thenumber of notes in the difference information of thetime-span trees that are included in TA but not in TA ⊓TB.
Step 2: Abstract notes in the difference information.
The note with the fewest dots in the difference information is selected and abstracted. The numbers ofdots can be acquired from the GTTM analysis results[1, 7]. If two or more notes share the fewest dots, weselect the first one reading the music left to right.
Step 3: Iterate.
Step 2 is iterated L times.
Subsumption relations hold as follows for the timespan trees TCm constructed with the above algorithm:

TA TB

4.2 Link of common information

4.3 Melody divisional reduction

TC

TD

TC
TD
Melody C

Melody D

TB

4.4 Combination of two melodies

TA
TC TD
Melody B
Melody A
A broken line in a time-span tree indicates that the branch has a special
operator [n1, n2].

Melody E

Figure 8. Overview of melody morphing method

TA

TB

TCn

TCn−1 … TC 2

TC1

TA .

(2)
In Fig. 8, nine notes are included in TA but not in TA ⊓TB.
Therefore, the value of n is 8, and we can obtain eight intermediate melodies Cm (m=1,2,…,n) between TA and TA⊓TB. Hence, melody Cm attenuates features that occur only
in melody A but not in B. Figure 9 illustrates this process.
In the same way, we can obtain melody D from TB andTA ⊓TB in the following manner:(3)
T A TBTD
TB .
4.4 Combination of two melodies
Finally, we use the join operator to combine melodies C
and D, which are the results of divisional reduction using
the time-span trees of melodies A and B. The simple join
operator is not sufficient for combining TC and TD because TC ⊔TD does not always exhibit monophony even if
TC and TD are monophonic. In other words, the result of
the operation has chords when the time-span structures
override, and the pitches of the notes are different; therefore, the result violates condition 4 listed at the beginningof Section 4: C will exhibit monophony if A and B are
monophonic.
To solve this problem, we introduce a special operator[n1, n2], which indicates either note n1 or note n2, as aresult of n1 ⊔n2. Then, the result of TC ⊔ TD is all monophonic combinations given by the operators.

5. MANUAL GENERATION OF INTERMEDIATE MELODY
Because of the problems described below with generatingintermediate melodies on a computer with the method
discussed in Section 4, we also had a musicologist manually generate a morphing melody.
5.1 Time-span analysis of two input melodies
Before linking the common information of two input
melodies, we need their time-span trees. Although wehave already developed grouping and metrical structure
analyzers based on deep learning [7], we have not yet
developed a high-performance time-span tree analyzer [4,6]. Acquisition of time-span trees thus requires manual
analysis by a musicologist.
If two input melodies are completely different, then
their common information will be empty, ⊥, making it
difficult to generate intermediate melodies. Therefore, thetwo input melodies must be carefully chosen by the musicologist. The musicologist selected a variation andtheme from Mozart’s "Twelve Variations" (K. 265/300e)
for the melodies, as explained in Section 6.
5.2 Time-span reduction
At present, the melody divisional reduction step of the

TA
LA＝1
LA＝2
LA＝3
LA＝4
LA＝5

LA＝6
LA＝7
LA＝8
TA TB

Figure 9. Melody divisional reduction

melody morphing method cannot be automated. This isbecause the height of each branch in the time-span tree is
not given, and the order of the notes to be reduced cannot
be determined. A musicologist who can analyze timespan trees, however, can easily generate reduced melodies by considering the inverse process of composition
depending on his/her interpretation, as described in Section 3.2. When multiple composition processes are possible, we consider all interpretations and then select one.
5.3 Selection of note from special operators
By selecting a note from each special operator after combining two melodies, the output melody will be monophonic when the two input melodies are monophonic.
Therefore, depending on how this note is selected, themelody changes. When the intermediate melody is unnatural, the musicologist adds notes to correct the melody.
The details of these added notes are described in the following section on the experimental results.

6. EXPERIMENTAL RESULTS
We asked a musicologist to generate intermediate melodies. The musicologist had over 10 years of experience in
GTTM analysis and a deep understanding of the melodymorphing method.

For the input melodies, the musicologist selected thetheme and Variation No. 1 of Mozart’s Twelve Variations
on "Ah vous dirai-je, Maman", K. 265/300e. She constructed nine intermediate melodies between the two
melodies. Figures 10 shows the first set of 8 bars for thetheme, Variation No. 1, and 9 intermediate melodies, outof 40 bars in total. In the figure, nonparenthetical notes
were generated with the melody morphing method,whereas parenthetical notes were added by the musicologist. For each melody, Table 1 lists the numbers of total
notes, notes generated with the melody morphing method,and notes added by the musicologist.
Notes were added in five ways.
a) Adding appoggiaturas, auxiliary notes, and passing notes.
b) Borrowing a note from a neighboring branch or
swapping the order of notes in a branch.
c) Dividing a note into two notes of the same pitch.
d) Expanding or contracting the melody.
e) Quoting a melody.
In Figure 10, morphing melodies 3 and 5 are examples of adding passing notes, denoted as (a), whereas (b)is an example of swapping the order of notes from Variation No. 1. In Fig. 10, (c) is an example of dividing a note
into two notes of the same pitch; (d) is an example ofexpanding a melody, specifically the last two notes in the
fourth bar of Variation No. 1; and (e) is an example of quoting a melody from the fifth and sixth bars of Variation No. 1.
All the notes in morphing melodies 1 and 2 were generated with the melody morphing method because both
melodies were close to the common information of thetheme and Variation No. 1, so melodies could be generated by simply choosing an appropriate note from eachspecial operator without adding any sound. All the notes
in morphing melody 5 were also generated with the melody morphing method because morphing melody 5 wasvery close to a melody ob-tained by divisional reductionof Variation No. 1, so all the notes were eighth notes.
Among morphing melodies 1, 2, and 3, the three melodies closest to the theme, only 1.5% of the notes wereadded by the musicologist. In contrast, morphing melodies 8 and 9, the closest to Variation No. 1, had 37.9% of
notes added by the musicologist. Notes needed to be add
ed to prevent unnatural melodies when generating multiple melodies between a melody with many eighth notes,
Table 1. Numbers of notes in each melody
Total number of notes

Theme
Morphing melody 1Morphing melody 2Morphing melody 3Morphing melody 4Morphing melody 5Morphing melody 6Morphing melody 7Morphing melody 8Morphing melody 9
Variation No. 1

82
51
51
93
121
94
157
176
253
267
271

Numberofnotes generated
with
melodymorphing
method51 (100%)
51 (100%)
90 (96.8%)
96 (79.3%)
94 (100%)
129 (82.2%)
157 (89.2)
164 (64.8%)
159 (59.6%)-

Numberofnotes added bymusicologist

0 (0%)
0 (0%)
3 (3.2%)25 (20.7%)
0 (0%)
28 (17.8%)
19 (10.8%)
89 (35.1%)
108 (40.4%)-

[2] G. Hadjeres, F. Pachet, and F. Nielsen, DeepBach: aSteerable Model for Bach Chorales Generation,
Proceedings of the 34th International Conference on
Machine Learning, PMLR 70:1362-1371, 2017.

such as morphing melody 5, and a melody with many
sixteenth notes, such as Variation No. 1.
In total, 78.5% of the notes were generated with the
melody morphing method, whereas the remaining 21.5%
were added by the musicologist.

[3] B. Sturm, J. Santos, O. Ben-Tal, and I. Korshunova.
Music transcription modelling and composition
using deep learning, CSMC2016, 2016.

7. CONCLUSIONS
We developed the Melody Slot Machine, which enables
users to manipulate melodies. For the Melody Slot Machine, we used our melody morphing method based on
the generative theory of tonal music (GTTM) for composition. We made melody morphing possible by combining
our melody morphing method with a manual process.
Experimental results were obtained by having a musicologist compose with the melody morphing method. In thecomposed pieces, 78.5% of the notes were generated withthe melody morphing method, while the remaining 21.5%
was added by the musicologist. We confirmed that thearranging processes used in the experiment could be classified into five types.
We plan to develop a support system for adjustmentusing the melody morphing method.

[4] M. Hamanaka, K. Hirata, and S. Tojo:“Implementing ‘A Generating Theory of Tonal
Music’”, JNMR, Vol.35, No.4, pp.249-277, 2007.
[5] M. Hamanaka, K. Hirata, and S. Tojo: “FATTA:
Full Automatic Time-span Tree Analyzer”,
ICMC2007, Vol.1, pp.153-156, 2007.
[6] M. Hamanaka, K. Hirata, and S. Tojo: “Sigma
GTTM III: Learning based Time-span Tree
Generator based on PCFG”, CMMR 2015, pp.303317, June 16-19, 2015.
[7] M. Hamanaka, K. Hirata, and S. Tojo:“DeepGTTMII: Automatic Generation of Metrical
Structure based on Deep Learning Technique,”
SMC2016, pp.221-249, 2016.

Acknowledgments
This work was supported by JSPS KAKENHI, Grant
Numbers 17H01847 and 16H01744.

[8] M. Hamanaka, K. Hirata, and S. Tojo, “Melody
Morphing Method based on GTTM,” ICMC2008, pp.
155-158, 2008.

8. REFERENCES[9] K. Hirata and S. Tojo, “Retrograde of Melody andFlip Operation for Time-Span Tree,” CMMR 2016,
pp.298-305, 2016.

[1] F. Lerdahl, and R. Jackendoff, A Generative Theoryof Tonal Music. Cambridge, Massachusetts: MIT
Press, 1983.
Theme

 

1





Morphing melody 1

 



Morphing melody 2

 

()

 




 






Morphing melody 4



 

Morphing melody 7

 





























( )







 

Variation No.1







Morphing melody 8

 





Morphing melody 6Morphing melody 9







Morphing melody 5

 





(a)



(b)












( )( ) ( ) (( ) ( ) ( ) (

)(

)(







)(
)

(d)





() ()


8








Morphing melody 3

 







7

6

5

4

3

2
















(  ) ( )






( )




( )



( ) ( ) ( ) 
(c)
(e) (
 
)
)
()
( ) ( )

Figure 10. First eight bars of theme, Variation No. 1, and nine intermediate melodies()