CONSTRUCTING MUSIC APPLICATIONS FOR SMARTPHONES
Masathoshi Hamanaka†, ‡Mikito Yoshiya‡

PRESTO, Japan Science and Technology Agency†
University of Tsukuba‡

ABSTRACTThis paper describes three applications for iPhone / iPod
Touch: ShakeGuitar, BandNavi, and Twitraker.
ShakeGuitar enables users to have a simulated experienceof playing a guitar by simply shaking the iPhone / iTouch;shaking with varying degrees of strength influences thelevel of guitar-melody difficulty, which smoothly changes
in real-time from soft backing to heavy soloing.
BandNavi enables users to find new bands and music byexamining the relations between musicians in bands,
which are affected by member changes, collaborations,etc. Twitracker enables users to share music fragments
by using twitter's time-line, and users can compose apiece of music using the fragments, even if they aremusical novices. The main advantage of our applications
is that they are based on our research on computer music.
So far, our applications have been downloaded morethan one hundred and fifty thousand times, and we havegotten a lot of positive comments from users.
1.

INTRODUCTION

Smartphones such as iPhone and Android include not
only telephone and e-mail functions but also functions to
execute a large variety of software applications. There
are many applications for iPhone and Android, and users
can easily download applications to these smartphones
from just about anywhere. Users can also easily develop
their own applications by using software development
kits for these phones.
Although many music applications have been released,
most of them have been sequencers [1], rhythm machines[2], and synthesizers [3], and they are very much like
software applications [1-3] that run on personal computers.
On the other hand, the Ocarina by Smule [4] enables
users to simulate playing the ocarina by touching a multitouch display and blowing on a microphone. Sound drop
[5] enables users to draw lines on a screen onto whichballs fall and make sounds that depend on the location of
the lines and the speed of the balls. These applications [4,5] exploit the advantages of smartphones, i.e., aconvenient multi-touch display and a microphone.
We have constructed three iPhone / iPod Touch
applications, called ShakeGuitar, BandNavi, andTwitracker. We constructed these applications with thefollowing purposes in mind.
 Investigations leading to practical applications
ShakeGuitar is intended to test the feasibility of the
melody morphing method we developed. The melody
morphing method generates an intermediate melody
between a melody and another melody with a systematicorder according to a certain numerical measure.
We have been working on an interactive melody
generator for composing melodies with the melody

Shyuhei Yoshida‡morphing method. However, users will not likely
understand what the melody morphing method is.
We developed ShakeGuitar as an easy way todemonstrate the melody morphing method.
 Evaluation and feedback by users
BandNavi is a music recommendation system whichenables users to discover new bands and music bytracing musicians who have participated in the same
band. Generally, it is difficult to evaluate theusability of a music recommendation system,
because we cannot evaluate the usability throughobjective measures such as the time taken by users tomanipulate the system. In other words, we need toevaluate systems subjectively as well as objectively.
We implemented an iPhone application andobtained information by means of questionnaires.
We hope to improve the method on the basis of the
feedback in the questionnaires.
 New interactions for creating music
Twitracker is a composing system, with which users
can post monophonic melodies on Twitter andcompose a song from the posted melodies.
Unfinished songs can be posted, shared, andmodified in the compositional process.
Twitracker represents a new sort of interaction
between users, because remote users can post andreuse a melody as simply as tweeting on Twitter, yet
it will be as if they are participating in a jam session.
We released ShakeGuitar, and so far more than onehundred and fifty thousand copies have been
downloaded from the iTunes App Store [6]. We feel that
ShakeGuitar has been useful to us in promoting the
melody morphing method. We also received a hundred
questionnaires on BandNavi; almost all respondents said
they had discovered a new band that they want to hear.
Moreover, users have said that they can collaborate byusing Twitracker to compose a piece of music.
2.

SHAKEGUITAR

ShakeGuitar (Figure 1) is a demonstration system forthe melody morphing method that changes the morphinglevel of each half bar by using the values from theaccelerometer in the iPhone / iPod Touch. When the

Figure 1. ShakeGuitar.

user stops moving the iPhone / iPod Touch, the unit
plays the backing melody of "The Other Day, I Met a
Bear (The Bear Song)". When the user shakes itvigorously, it plays heavy soloing. When the user shakes
it slowly, it plays a morphed melody between thebacking and the heavy soloing.
2.1. Time-Span Tree of GTTM
Our melody morphing method uses time-span trees
acquired by analysing the results of the generative theoryof tonal music (GTTM) [7]. Figure 2 is an example ofabstracting a melody by using a time-span tree. There is atime-span tree from melody D, which embodies the resultsof the GTTM analyses. The important notes are connected
to a branch nearer the root of the tree. In contrast,
unimportant notes are connected to the leaves of the tree.
We can obtain an abstracted melody E by slicing the tree
in the middle and omitting notes that are connected tobranches under line E. In the same manner, if we slice thetree higher up at line F, we can get a more abstracted
melody F. We regard the abstraction of a melody as a kindof melody morphing because melody E is an intermediate
melody between melody D and melody F.

2.3. Melody Morphing Method
By using the time-span trees TA and TB from melodies A
and B, we can calculate the most common information,
TA ⊓ TB, which are the essential parts of melody A, as
well as those of melody B. The meet operations TA ⊓ TB
are abstracted from TA and TB, and those discarded notes
are regarded as the difference information of TA and TB(Figure 4a). We consider that there are features without
the other melody in the difference information of TA and
TB. Therefore, we need a method for smoothly increasingor decreasing these features. The melody divisional
reduction method can abstract the notes of the melody inthe differential branch of the time-span tree [9]. We use
the join operator to combine melodies C and D, which are
results of the divisional reduction or augmentation using
the time-span tree of melodies A and B (Figure 4c).
TA TB

TC

TD

（a）Linking the common information of melodies
TC

Melody C

LA （b）Melody division
reduction

Melody D

TD
LB

TA

（c）Combining
two melodies

TB
TC

TD

Melody A

Melody B

Melody E

Figure 2. Abstraction of melody.

2.2. Primitive Operation Using Time-Span Tree
In melody morphing, we use the primitive operations of
the subsumption relation (written as ⊑), meet (written as⊓) and join (written as ⊔), as proposed by Hirata [8].
The subsumption relation represents the relation "aninstantiated object" ⊑ "an abstract object" (Figure 3a).
For example, the relationships among TD , TE and TF,
which are the time-span trees (or reduced time-spantrees) of melodies D, E and F in Figure 2, can be
represented as follows:
TF ⊑ TE ⊑ TD
The meet operator extracts the largest
common part,or the most common information of the time-spantrees of two melodies in a top-down manner (Figure3b). The join operator joins two time-span trees in the
top-down manner as long as the structures of twotime-span trees are consistent (Figure 3c).
(a)TA(b) TBTA(c) TB⊑
Instantiating Abstracting

TA ⊓ TBTA ⊔ TB

Figure 3. Examples of subsumption ⊑ meet ⊓ and join ⊔ .

Figure 4. Overview of melody morphing method.

3.

BANDNAVI

The previous music retrieval systems for audio signal ofenable users to retrieve similar music, but they cannot be
used to look for music related to a particular musician
because they only use audio signal similarity orcollaborative filtering. On the other hand, BandNavi is able
to discover various bands in which a musician has
participated by using member change information collected
automatically from the web.
3.1. The Band-Network
A musician often has the relations with more than one
band he or she may be a guest musician and or havebeen a member in a number of bands. Therefore,
relationships between musicians and bands form acomplex network connected by lines. We call the
network formed by connecting an affiliation relationshipof bands and musicians in the Band-Network (Figure 5).
3.2. Using Web Mining to Acquire Members’ Names
We constructed the Band-Network by automatically acquiringof band-member names from Web pages. The member
name extraction process is composed of four phases.

Neil Murray
Ian PaiceGarry Moore
Simon Phillips
Jimmy Bain

Whitesnake

Cozy Powell

Quiet Riot
Rudy Sarzo

Jon Lord

Don Airey

MusicianBand

Steve Vai

Garry MooreDavid
Coverdale

Ozzy Osbourne

Rainbow
Bob Daisley
Mickey Lee Soule
Ritche Blackmore

Tommy Aldridge

Deep Purple

Ian Gillan
Joe Lynn Turner

Ozzy Osbourne

Figure 5. Band-Network.

First, we use Google’s Search API to collect Web pages.
Given a band name B, we retrieve Web pages using by thequery that adds the following keywords after B. (Thecharacter “+” in the keywords means “AND”.)
A)“B”+music
B)“B”+bandC)“B”+band+member
D)“B”+band+lineup
Second, we detect the person names in the collected
pages by using N-grams from the plain text. We assume
that most names are composed from two words up to
four words; therefore, an N of 2,3, or 4 is used.
Third, we extract the musician names by patternmatching with the instrument name and person name set.
We extract the musician names with this clue because amusician name often appears on the Web page together
with the name of the instrument which he or she plays(guitar, drum). We use the following three patterns.
a)Role Memberb)Member (.*Instrument.*)c)
Member -.*Instrument.*
Here, the Instruments of b) and c) are names such as“guitar” and “drums” used by the band. Accordingly, theRoles of a) are names such as “guitarist” and “drummer”.
The person names which match any patterns of a, b, and c on
the collected Web pages are regarded as musicians’ names.
Finally, we automatically remove unwanted elements,
i.e., the names of persons who aren’t band members.
3.3. Implementation

the list of bands that Roger Glover participated in wasdisplayed (Figure 6b). After that, a user selected theband “Rainbow”, and the member list of Rainbow was
displayed (Figure 6d).
When we push the button to the right of the band
name in the navigation mode, The mode changes to themedia access mode and two buttons, “Search In
YouTube” and “Search In iTunes Store”, are displayed(Figure 6e). By pushing the buttons, we can trial listen
through YouTube or the Samples in the iTunes Store.
In browse mode, we can verify the relationshipbetween a band and a member by browsing the web
page that the relationship was extracted (Figure 6i). We
can also correct incorrect information.
4.

Twitracker is an iPhone / iPod Touch application inwhich user can collaborate in composing music throughTwitter. User can record a fragment of a melody byusing the keyboard displayed on the iPhone / iPod
Touch and post the fragment on Twitter (Figure 7a, 7b).
4.1. Posting Melody Fragments on Twitter
In order to post a melody fragment on Twitter, we use
Music Macro Language (MML) [10], because MML
expresses a music score in text format. We also use the
URL shortener service [11], because Twitter only
accepts text no more than one hundred and forty(a) Keyboard(i) List of extracted URL (h) Option in member
nameFavorite(b) Data list (local)(d) Timeline

Favoritelist

Figure 7. Overview of Twitracker.
Media access mode

Navigation mode(a) Member list of Band(c) Twitter client (online)

RecordBandNavi has three modes: navigation, media access,and browse (Figure 6).
In navigation mode, we can trace musicians by
iteratively displaying the member list of a band and theband list of a member. For example, in a test when we
played a song by “Deep Purple”, the member list of
Deep Purple was displayed (Figure 6a). Then, a user
selected the musician “Roger Glover” from the list, andBrowse mode

TWITRACKER(e) Option in band (f)iTunes Music Store
name(b) Band list of a memberAB(c) Song list inmusic libraryC(j) Browsingof extracted page(d) Member list of Band

Figure 6. Overview of BandNavi.

(g)YouTube

characters long and songs written in MML sometimes
exceed this limit. The URL shortener service reduces thesize of MML text and enables it to be posted on Twitter.
The users can listen to fragments that another user
has posted by using the Twitter client on Twitracker(Figure 7c). On each tweet, there is a play button forplaying the fragment and a favorite button for putting thefragment on the favorites list.
4.2. Composing using Twitracker
Figure 7d is a screen snapshot of the editing window forcomposing a piece of music. On the right side of theediting window, there is a favorites list of fragments inthe form of Twitter icons of the composer-user. If there
are more than two fragments from the same user on thelist, color of the outline of the icon will change.
In the middle of the editing window, there is atimeline for dropping the icon from the favorites list. In
the timeline, the horizontal axis indicates the time axis
and vertical axis indicates the part.
5.

DISCUSSION

The three iPhone / iPod Touch applications each havetheir own purpose. In this section, we discuss the level towhich these apps have achieved their purposes.
5.1. Figures, tables and captions
ShakeGuitar was downloaded more than one hundredand fifty thousand times. Figure 8 is the download
ranking of iTunes App Store (Japanese) in the music
category. Shakeguitar reached third place. ShakeGuitar
was featured on a TV program, and hence we can saythat it is quite popular.
Thus, as a result of releasing this app, many people
have come know the nature of the melody morphing
method. We therefore believe that ShakeGuitar has beena very successful means of promoting the method.
2009/10 2009/12 2010/01 2010/03 2010/05 2010/06 2010/08 2010/09 2010/11 2011/01
0
10
20
30
40
50
60

Figure 8. Download ranking of ShakeGuitar.

5.2. Feedback from Users
We prepared questionnaires for BandNavi and posted
them at the following URL: http://musicdb.iit.tsukuba.
ac.jp/bandnavi.html. So far, we have gotten more than ahundred response from users. The questionnaires
indicated almost all respondent had discovered a bandthat they hadn’t heard of before and they enjoyed tracing
the relations between bands and musicians.
Seventy percent of respondents were able tounderstand the operation method of BandNavi within
five minutes. However, five percent needed more thanfifteen minutes to understand the operation method. We

plan to investigate the reason why some people took
such a long time to understand, and we should
implement changes, if necessary.
5.3. Creating a New Means of Interaction
Although many collaborative composing system havebeen proposed [11], Twitracker enables composers toshare fragments of melodies thorough Twitter andcollaborate in composing music. We hope that Twitracker
will lead to a new style of composition in which musicians
compose a piece of music by sharing their work.
6.

CONCLUSION

We developed three iPhone / iPod Touch applications.
Of them, ShakeGuitar was downloaded more than one
hundred and fifty thousand times, and it has proven to bea good way of promoting the melody morphing method.
BandNavi recommends new songs or new artists bystoring relationships between bands and musicians.
More than a hundred users filled out or ourquestionnaire on the usability of the system, and theiranswers helped us to improve the system. Twitracker
enables people to collaborate in composing music byusing Twitter. We plan to study how to guard against
infringement of copyright when using Twitracker.
7. REFERENCES[1] Racjin, T. ''iPhrase'', http://www.touchracjin.com/wp/, 2010.
[2] Mitchell, T. ''Instant DJ'', http://toastkitten.webs.com/, 2010.
[3] Yoshida, S., and Newforester Inc. ''NESynth''.
http://www.newforestar.com/nesynth/index.html, 2010.
[4] Wang, G., G. Essl, J. Smith, S. Salazar, P. Cook, R.
Hamilton, R. Fiebrink, J. Berger, D. Zhu, M.
Ljungstrom, A. Berry, J. Wu, T. Kirk, E. Berger, J.
Segal. “Smule = Sonic Media: An Intersection of the
Mobile, Musical, and Social”. Proceedings of theInternational Computer Music Conference, pp. 283286, 2009.
[5] Develoe LLC., ''Soundrop'', http://develoe.com/, 2010.
[6] Apple Inc., ''iTunes App Store'',http://itunes.apple.com/app/, 2010.
[7] Lerdahl, F., and Jackendoff, R. A Generative Theory ofTonal Music. Cambridge, Massachusetts: MIT Press, 1983.
[8] Hirata, K., and Aoyagi, T. '' Computational Music
Representation Based on the Generative Theory of Tonal
Music and the Deductive Object-Oriented Database'',Computer Music Journal, 27:3, pp. 73-89, 2003.
[9] Hamanaka, M., Hirata, K., and Tojo, S. ''Melody
Morphing Method based on GTTM'', Proceedings ofthe International Computer Music Conference, pp.
155-158, 2008.
[10] LLC books Inc. Music Notation File Formats: Gnu
Lilypond, General MIDI, ABC Notation, Music
Macro Language, Impro- Visor, Open Sound
Control, ASCII Tab, 2010.
[11] Google Inc., Google URL Shortener, http://goo.gl/ ,2010.
[12] Hirata, K., Kaji, K., and Nagao, K. ''AnnotatedMusic for Retrieval, Reproduction, and Sharing'',
Proceedings of the International Computer Music
Conference, pp. 584-587, 2004.