Proc. of the 16th International Symposium on CMMR, Tokyo, Japan, Nov. 13-17, 2023

Melody Changing Interfaces for Melodic Morphing
Masatoshi Hamanaka1
RIKEN masatoshi.hamanaka@riken.jp

Abstract. We have developed several applications based on the Generative Theory of Tonal Music utilizing the melodic morphing method. Since multiple melodies
generated by the morphing method have similar musical structures, the global
structure of the melodies does not change when a portion of one melody time
axis is replaced by another. When developing the apps, we used dial-based andgrid-based interfaces for switching melodies. In this paper, we present the resultsof a comparison of the two interfaces conducted with 30 users.
Keywords: Melody switching interface, Melodic morphing method, Generative
Theory of Tonal Music (GTTM), Dial-type interface, Grid-type interface

1Introduction

We have developed several applications using a melodic morphing method based on theGenerative Theory of Tonal Music (GTTM) [1, 2]. In the GTTM, a time-span tree is abinary tree in which each branch is connected to each note (Fig. 1). The branches ofa time-span tree are connected closer to the root than those connected to structurally
important notes.
The main advantage of time-span trees is that they can be used to reduce notes.
Specifically, reduced melodies can be extracted by cutting a time-span tree with a horizontal line and omitting the notes connected below the line. In melody reduction with
GTTM, these notes are essentially absorbed by structurally more important ones.
We previously proposed a melody-morphing method that applies this reduction(Fig. 2) to generate a melody that is structurally intermediate between two input melodies[3,4]. This is done by combining two melodies after executing the reduction on their respective time-span trees.
Since multiple melodies generated by the morphing method have similar musicalstructures, the global structure of the melodies does not change when a portion of one
melody time axis is replaced by another. When developing apps with the melodic morphing method, we used dial-based and grid-based interfaces for switching melodies. In
this work, we present the results of a comparison of the two interfaces conducted with
30 users.

This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).

772

Proc. of the 16th International Symposium on CMMR, Tokyo, Japan, Nov. 13-17, 2023

Melody C︙ (c) Combining two melodies

σα σβσA
Melody A

Time-span tree

σα(b) Partial melody reductioni

Melody ασBσβMelody Bj

σA σB

Melody β(a) Linking common events

Fig. 1. Time-span tree.

2

Fig. 2. Melodic morphing method.

Applications for Melodic Morphing Method

ShakeGuitar The ShakeGuitar (Fig. 3(a)) and ShakeGuitarHD (Fig. 3(b)) apps changethe morphing level according to the speed at which the iPhone or iPad is shaken [5].
For the morphing input, we utilize the basic melody of “The Other Day I Met a Bear”
and the melody of a guitar solo played with the same chord progression. The uniquefeature here is that not only interpolation of the two melodies but also extrapolation isperformed [6]. With the extrapolation, we can generate an intense guitar solo with morenotes than the original. When the iPhone is held stationary, a basic melody is played,
and the faster the iPhone is shaken, the more intense the melody becomes.
ShakeGuitar and ShakeGuitarHD both feature a grid mode with time on the verticalaxis and morphing level on the horizontal axis. The morphing level can be changed bytouching the grid. In ShakeGuitarHD, the guitar is animated to swing up and down, andthe width of the swing changes according to the morphing level. The morphing levelalso changes depending on how fast you swipe your finger up and down on the swinging
guitar.
Melody Slot Machine We developed the Melody Slot Machine, a research demonstration device, to promote the melodic morphing method. With this application, theperformer’s movements can be viewed on a Pepper’s ghost display (Fig. 4(a)). Melody
segments are displayed on a dial, and the melody to be played can be switched by rotating the dial (Fig. 4(b)). We exhibited the Melody Slot Machine at an international conference shortly after it was developed, but the COVID-19 pandemic made it difficult toconduct further demonstrations in person [7–9]. We therefore adapted the Melody Slot
Machine for the iPhone so that people could experience it simply by downloading theapp.
Melody Slot Machine for iPhone Figure 5 shows a screenshot of the Melody Slot
Machine iPhone app [10]. The horizontal axis is time, and each dial displays a melody
segment in musical notation (Fig. 5(a)). By swiping up and down on each dial, you canswitch between the segments. Due to the limited screen size of the iPhone, only four
melody segments can be viewed simultaneously, and the currently playing segment canbe viewed by automatically scrolling left as the musical piece progresses (Fig. 5(b)).

773

Proc. of the 16th International Symposium on CMMR, Tokyo, Japan, Nov. 13-17, 2023

(a) Pepper's ghost
display(a) ShakeGuitar (b) ShakeGuitarHD(b) Slot dialFig. 3. Screenshot of ShakeGuitar.

Fig. 4. Melody Slot Machine.

The dial changes for the entire musical piece are displayed in a grid at the bottomof the screen, corresponding to the numbers written on the dials . Swiping up from thebottom of the grid display brings up the full-screen grid (Fig. 5(c)), and users can touch
it to change the selected grid. The change in the grid is linked to the dial, and the melody
is played reflecting the change. Swiping down terminates the full-screen grid, and thedial appears again. When the iPhone is shaken up and down, each dial is shuffled togenerate a new combination of melodies (Fig. 5(d)).
Melody Slot MachineHD Figure 6(a) shows a screenshot of Melody Slot MachineHD,in which the symbols represent changes in melody variations [11]. For example, themusical note symbol means that the same variation will continue, and the cherry symbol
indicates that the variations will change one after another.
Pressing the mode-switch buttons on the left and right of the screen displays the gridtile screen, and you can check and change the variations in the entire song (Fig. 6(b)). If
you use the grid to change the combination of variations, the symbols on the slot screen
will also change accordingly.
The performer screen is displayed by pressing the mode-switch buttons or holding
the iPad vertically (Fig. 6(c)). This display shows a performer playing new combinations of melodies determined by the slots or grids. Short interpolation video clips of theperformer generated by AI are sandwiched into recorded videos of an actual performer,
so the performer moves seamlessly.
(a) Dial display(b) Scrolling left(c) Full-screen grid(d) Dial shuffled

Fig. 5. Melody Slot Machine iPhone app.

774

(a)

(c)

(b)

(d)

Fig. 6. Melody Slot MachineHD.

Proc. of the 16th International Symposium on CMMR, Tokyo, Japan, Nov. 13-17, 2023

3

Experimental Results

We launched the Melody Slot Machine iPhone app in May 2021 and Melody Slot MachineHD in March 2022. As of July 2023, they have downloaded 657 times. Afterlaunching the app ten times, users are presented with a message inviting them to complete a questionnaire regarding its usability. The following is a portion of the questionnaire.
Q1: Which was easier to operate, the dial screen or the grid screen?
Q2: Which was more enjoyable to operate, the dial screen or the grid screen?
Thirty responses were received, 20 people said that the dial type was easier to operate and more enjoyable than the grid type.

4

Conclusion

In this work, we compared several interfaces that change the melody of applications
using a melodic morphing method based on the Generative Theory of Tonal Music.
Our findings showed that more people found the dial-type interface easier and moreenjoyable to operate than the grid-based one.
In the app version of the Melody Slot Machine, the dials on the iPhone were musicalnotations and on the iPad they were symbols, but we plan to make it possible to switch
between the two types of dials on both devices.

References
1. Lerdahl, F. and Jackendoff, F.: A Generative Theory of Tonal Music. The MIT Press, Cambridge, MA (1983).
2. Hamanaka, M., Hirata, K., and Tojo, S.: Implementing “A Generative Theory of Tonal Music”.
Journal of New Music Research, 35(4), 249–277 (2006).
3. Hamanaka, M., Hirata, K., and Tojo, S.: Implementation of Melodic Morphing based on Generative Theory of Tonal Music. Journal of New Music Research, 51(1), 86–102 (2022).
4. Hamanaka, M., Hirata, K., and Tojo, S.: Melody Morphing Method based on GTTM, in Proceedings of the 2008 International Computer Music Conference (ICMC2008), pp. 155–158(2008).
5. Hamanaka, M., Yoshiya, M., and Yoshida, S.: Constructing Music Applications for Smartphones, in Proceedings of ICMC2011, pp. 308–311 (2011).
6. Hamanaka, M., Hirata, K., and Tojo, S.: Melody Extrapolation in GTTM Approach, in Proceedings of ICMC2009, pp. 89–92 (2009).
7. Hamanaka, M., Nakatsuka, T., and Morishima, S.: Melody Slot Machine, ACM SIGGRAPH
2019 Emerging Technologies ET-245, 2 pages (2019).
8. Masatoshi H.: Melody Slot Machine: A Controllable Holographic Virtual Performer, in Proceedings of the 27th ACM International Conference on Multimedia (MM’19), pp. 2468–2477(2019).
9. Nakatsuka, T., Hamanaka, M., and Morishima, S.: Audio-guided video interpolation via human pose features, in Proceedings of VISIGRAPP2020, pp. 27–35, (2020)
10. Hamanaka, M.: Melody Slot Machine on iPhone, Submitted ACM Symposium on User Interface Software and Technology (UIST’23), 2 pages (2023).
11. Hamanaka, M.: Melody Slot Machine HD, ACM SIGGRAPH 2023 AppyHour, 2 pages(2023).

775