IPSJ Journal

Vol. 50

No. 122923–2936 (Dec. 2009)

Regular Paper

Musicream: Integrated Music-Listening Interface for
Active, Flexible, and Unexpected Encounterswith Musical Pieces
Masataka Goto†1 and Takayuki Goto†1,∗1
This paper describes Musicream, a novel music-listening interface that lets auser unexpectedly come across various musical pieces similar to those liked by
the user. Most existing “query-by-example” interfaces are based on similaritybased searching, so they return the same results for the same query, meaningthat a user of those systems always receives the same list of musical pieces
sorted by similarity. Therefore, most existing systems do not provide a user an
opportunity to encounter various unfamiliar musical pieces by chance. Musicream facilitates active, ﬂexible, and unexpected encounters with musical pieces
by providing four functions: the music-disc streaming function which createsa ﬂow of many musical-piece entities (discs) from a large music collection, thesimilarity-based sticking function which allows a user to easily pick out andlisten to similar pieces from the ﬂow, the meta-playlist function which cangenerate a playlist of playlists (ordered lists of pieces), and the time-machinefunction which automatically records all Musicream activities and allows a user
to visit and retrieve a past state as if using a time machine. In our experiments,
these functions were used seamlessly to achieve active and creative querying andbrowsing of music collections, conﬁrming the eﬀectiveness of Musicream.

1. Introduction
Enterprise services for delivering music via the Internet are increasingly
widespread and continued growth in the demand for such services seems inevitable in the foreseeable future. Some advanced services provide ﬂat-rate, unlimited music subscription services that give users unlimited on-line access toseveral million musical pieces 1 . The most popular and basic interface for thoseservices is based on lists of bibliographic text information such as titles, artist†1 National Institute of Advanced Industrial Science and Technology (AIST)∗1 Presently with The Graduate University for Advanced Studies2923

names, and genres provided on a display. It typically provides a function forplaying back musical pieces in order and a function for searching musical pieces
or artists by specifying bibliographic information. It also provides a function forbrowsing the whole music collection by choosing genres and a function for recommending musical pieces or artists. This text-based interface works well whena user already has in mind particular musical pieces that he or she would like tolisten to. However, given that such music services oﬀer a huge music collection,
many users tend to mainly listen to familiar musical pieces for a few months
and then become bored with those pieces. Thus, users will increasingly feel desires like “I want to hear something diﬀerent that suits my taste” and will want
to ﬁnd something new by themselves when accessing ﬂat-rate, unlimited music
subscription services or large personal music collections. In addition, declining
production costs have made it much easier for amateur or professional artists to
produce their own music without the support of record labels and the Internet
enables such artists to easily reach out to their audiences; as a result, personalmusic collections full of unfamiliar pieces are becoming bigger and bigger 2 .
The goal of our study is to provide a comprehensive integrated music-listening
environment in which various browsing and querying functions can be used inan easy and seamless manner to increase opportunities to encounter unfamiliar but interesting musical pieces when actively browsing a musical collection.
Although various previous studies have resulted in music-recommendation or
playlist-generation schemes 1)–8) based on collaborative (social) or content-based
ﬁltering, encounters with unfamiliar music tend to be passive on the part of listeners. While many eﬀective similarity measures 9)–16) have been proposed to enable users to listen to music of similar mood, most previous “query-by-example”
interfaces for searching similar music only provide functions for listing musicalpieces similar to a certain musical piece. Because a user will always receive thesame ranked list of musical pieces given the same query and music collection, op1 For example, Napster 2.0, a popular online music subscription service, lets users listen to
over eight million musical pieces for a monthly ﬂat-rate subscription fee. It had over 830,000
paid subscribers in 2007.
2 Nowadays, for example, many amateur artists submit their musical pieces to video sharing
services.

c 2009 Information Processing Society of Japan


2924

Musicream: Integrated Music-Listening Interface for Active, Flexible, and Unexpected Encounters with Musical Pieces

portunities to encounter previously unknown musical pieces in the collection arelimited. As for the music playback order, many music players already have functions enabling users to specify playlists (i.e., lists of musical pieces for playback).
The drawback here is that the operations provided by these functions are usuallylimited to changing the order of pieces and to inserting and deleting pieces inexisting playlists. In addition, the use of each playlist is often temporary, wherea created playlist is discarded after being used several times. While it might beinteresting for a user to listen to music from a past playlist (e.g., from ten years
earlier), this is impossible unless the user periodically saves playlists, which isquite troublesome.
To overcome these limitations, we have developed a music-listening interface
called Musicream (“music” + “stream”) that enables a user to initiate unexpected
but interesting encounters with musical pieces and provides a user with ﬂexible
interfaces for actively browsing a music collection and managing playlists. The
idea behind Musicream is to see if we can break free from stereotyped ideasof how music-listening interfaces must be based on lists of musical-piece titles
and artist names. Musicream applies the following functions to create a novel
music-listening environment without listing musical-piece titles.
( 1 ) Music-disc streaming function: lets musical-piece entities (discs) stream
down one after another on a screen for user browsing and selection. Thisfacilitates unexpected favorable encounters when the user is actively browsing part of a collection consisting of several million musical pieces: givensuch a huge collection, it is diﬃcult to visualize and manage the entire
collection on a screen.
( 2 ) Similarity-based sticking function: attaches similar musical pieces to a selected music disc like pieces of metal stick to a magnet 1 . This is an active
and ﬂexible version of the useful “query-by-example” similarity search.
( 3 ) Meta-playlist function: enables the user to try out various playback orders
by making it easy to rearrange groups of musical pieces on the screen ina manner similar to rearranging CDs on a real-world desktop. This helps
the user devise better playlists.
1 The use of a magnet metaphor for an interface is also discussed in Ref. 17).

IPSJ Journal

Vol. 50

No. 122923–2936 (Dec. 2009)( 4 ) Time-machine function: provides a means of moving freely back in time torestore a past Musicream screen as if using a time machine to return to apoint in the past where one was listening to music. This is a version of theuseful “undo” function that provides the ultimate in ﬂexibility.
These functions make it possible, for example, to pick out a musical piece (disc)from among many streaming on the screen and then attach other pieces similar
in mood to that musical piece, thereby creating a group of pieces. Furthermore,by simply continuing this process, several groups (playlists) of musical pieces canbe formed and placed in a blank area on the screen in a manner that determines
the playback order of those groups. In this way, users can enjoy music in amuch more active manner. Moreover, all operations on Musicream are recorded
automatically so that any past state can be returned to at any time. A user can
reproduce what he or she was listening to at some point in the past and thencontinue with other operations from within that state if so desired.
This paper is organized as follows. Section 2 introduces the functions provided
by Musicream and Section 3 describes how they are implemented. Section 4
presents the results of experiments on the use of Musicream. Section 5 discusses
related research and Section 6 summarizes this paper’s contributions.
2. Musicream Functions
Musicream provides four novel functions (Fig. 1) to satisfy user desires like “Iwant to hear something diﬀerent that suits my taste.” Instead of working as
independent functions, they operate in a cooperative and comprehensive manner
to enable new music-listening formats. Each of these functions is described below.
2.1 Music-Disc Streaming Function
This function presents the user with images of discs, each corresponding toa musical piece in a music collection 2 . These discs ﬂow from top to bottomon the screen, one after another, and the user can select a disc and listen tothat musical piece. This function is especially useful as a means of encounteringvarious musical pieces by chance as opposed to making a speciﬁc request as in “I2 In this paper, “music collection” means any set of musical pieces that a user is able tolisten to. This would include groups of musical pieces stored on portable music players andpersonal computers as well as those on ﬂat-rate, unlimited music subscription services.

c 2009 Information Processing Society of Japan


2925

Musicream: Integrated Music-Listening Interface for Active, Flexible, and Unexpected Encounters with Musical Pieces(a) Music-disc streaming function: Active and unexpected encounterswith interesting musical pieces(b) Similarity-based sticking function: Controllable and ﬂexible“query-by-example” similarity search(c) Meta-playlist function: Advanced playlist editing using a playlist of
playlists(d) Time-machine function: Intuitive browsing and perfect recall of
past music listening activities

Fig. 1 Four functions of Musicream.

want to hear this musical piece.”
The encountering of previously unknown music by chance is not without precedent. People often listen to music broadcast on radio and television or listen to
songs or musical pieces that appear on hit charts or are recommended by friends.
Various studies have been made on music-recommendation or playlist-generation
schemes 1)–8) based on collaborative (social) or content-based ﬁltering. If a person were to listen to only musical pieces that have been selected by other people
or systems, though, such encounters would tend to be passive from the user’s

IPSJ Journal

Vol. 50

No. 122923–2936 (Dec. 2009)

point of view. At the same time, it is diﬃcult to use a personal music collection
consisting of thousands of musical pieces for the purpose of encountering musicby chance. While music search methods based on bibliographic information suchas titles and artist names have been used, they have not considered chance encounters with musical pieces 1 . There are also methods based on folder-based
1 Although some existing players support a random shuﬄe playback function (such as Apple
iPod shuﬄe), they do not allow a user to intentionally control the general nature of musicencounters, which Musicream does allow.

c 2009 Information Processing Society of Japan


2926

Musicream: Integrated Music-Listening Interface for Active, Flexible, and Unexpected Encounters with Musical Pieces

Fig. 2 Music-disc streaming function: Discs corresponding to musical pieces stream
downwards from three colored taps.

hierarchical classiﬁcation by music genre, artist name, etc. that allow a user tosearch from one hierarchical layer to another to reﬁne the search. These methods, however, narrow down searches by promoting hierarchical selections such as“jazz ⇒ bebop,” which unnecessarily decrease the possibility of unexpected but
interesting encounters with musical pieces.
Our music-disc streaming function enables music encounters not likely to occur
through broadcasting, recommendations, traditional searching, or hierarchicalstructures. Figure 2 illustrates the Musicream screen along with some basic
terminology. Each of the three rectangular boxes at the top represents a musicsupply “tap” that releases small discs one at a time, each disc corresponding toa musical piece. These discs fall straight down at the same speed (since a tap
is fairly wide, released discs are spread out crosswise along a tap). The taps aredesigned so that each releases musical pieces of a diﬀerent mood, and the rateof release — i.e., the interval between disc releases from the same tap — can be
adjusted with the sliders above each tap. A falling disc that reaches the bottom of
the screen (the “ground”) disappears. The user may remove a disc that appears
interesting from these streams of falling discs and listen to that musical pieceby dragging the disc using a mouse (or a stylus, touch tablet, etc.). Rolling themouse pointer over a small disc expands it and displays the title and name of theartist for the musical piece in question, as shown in the ﬁgure to the left.
The beneﬁt of this interface is that a user can feel that music encounters areintentionally controlled. If one of several million musical pieces is randomly
selected and presented to the user, the user would probably not feel any particular

IPSJ Journal

Vol. 50

No. 122923–2936 (Dec. 2009)

interest in it. On the other hand, with the music-disc streaming function theuser deliberately chooses a disc from the streaming discs, and so is more likely
to feel interest in that piece of music and listen to it. We think such active andintentional interaction can change user behavior in this way.
Each tap and disc is given a color that reﬂects the mood or feeling of a musicalpiece and each disc falls out of a tap with a similar color. In other words,
similarity in color is associated with similarity in musical pieces, which meansthat a user who likes the musical piece of a selected disc can choose other pieces
based on the color of that disc. While using the music-disc streaming function, auser will often want to perform that operation time after time, and this is where
the similarity-based sticking function described next comes in.
2.2 Similarity-Based Sticking Function
With this function, the user picks up a disc previously removed from the streaming discs and drags it to touch other discs that are still streaming. The original
disc can be dragged while being played back without interruption. This operation has the eﬀect of selectively “sticking” (attaching) discs of similar mood tothe original disc one after another. This function may be viewed as a “query-byexample” search for musical pieces, but it is not a search that simply presents alist of similar songs. In our similarity-based sticking function, the user collects
musical pieces from streaming pieces according to the user’s own choice. The
important point here is that musical pieces can be encountered while the user
is performing an operation similar to the way people pick up things they like inreal life.
Here, the “stickiness” has been designed to depend on the similarity between
two musical pieces in terms of mood (timbre similarity) 1 . “Stickiness” means
the ease of sticking a disc from the streaming discs. As shown in Fig. 3, twodiscs with high similarity will stick on ﬁrst contact (high stickiness) whereas twodiscs with low similarity will not stick unless brought into contact several times(low stickiness). In this way, the range of similarity of musical pieces to be added
can be easily adjusted by appropriately maneuvering the picked-up disc through1 Instead of mood, which is used in our current implementation, any state-of-the-art similarity
based on content-based or collaborative ﬁltering can be used for this function.

c 2009 Information Processing Society of Japan


2927

Musicream: Integrated Music-Listening Interface for Active, Flexible, and Unexpected Encounters with Musical Pieces

Fig. 3 Similarity-based sticking function: The “stickiness” depends on similarity in terms ofmood. Similarity in color is associated with similarity in musical pieces.

Fig. 5 Editing operations for a disc series (playlist).

Fig. 4 Two disc-series modes.

the streams of falling discs.
Sticking discs together using this function forms an overlapping series of discs
as shown on the left of Fig. 4. This is called “compact mode.” Clicking the top
disc of this series rearranges the discs into a non-overlapping, horizontal series asshown on the right of Fig. 4. This is called “maintenance mode.” In this mode,
moving the mouse pointer over a disc expands that disc to make it the focus ofthat series (the discs to either side of the disc-in-focus become slightly larger).
It is also possible to rearrange the disc order in the series or remove a disc fromthe series, as shown in Fig. 5.
In maintenance mode, the disc components shown in Fig. 6 appear, thus enabling the following functions.
• Playback control
Clicking on the playback button on the disc starts playback of that musicalpiece. If another musical piece is being played back, it is stopped rightafter this click (the start of playback). During playback, an animated ripple
eﬀect emanates from the disc. If the piece is played until its end, playback
automatically changes to the next underlying disc. Clicking again on the
button stops the playback.
• Playback position control

IPSJ Journal

Vol. 50

No. 122923–2936 (Dec. 2009)

Fig. 6 Disc components.

The playback position slider, whose function is the same as that of playback position sliders provided by ordinary music players, is placed along theperiphery (circumference) of the disc. One complete loop in the clockwise
direction from the twelve o’clock position corresponds to the length of thatmusical piece. Clicking on any position along the slider starts playback fromthat position in the piece.
• Minimize
Clicking on the minimize button at the center of the disc switches to compact
mode.
• SortClicking on the sort-mode button on the disc switches to sort mode as shownin Fig. 7. Using the disc-in-focus as a base for sorting, a copy of the otherdiscs in the series appears below that disc in descending order of mood similarity. These duplicated discs disappear upon exiting sort mode.
In Musicream, such a disc series is already a highly functional playlist (i.e., alist of musical pieces that speciﬁes playback order). In existing music players,“zapping” (changing from one musical piece to another in a playlist as the urge

c 2009 Information Processing Society of Japan


2928

Musicream: Integrated Music-Listening Interface for Active, Flexible, and Unexpected Encounters with Musical Pieces

Fig. 7 Sort mode: Using the disc-in-focus as a base, other discs in the series descend inorder of mood similarity.

arises) in music playback requires the user to perform a two-step procedure. First,
the user must (double-)click the title of the desired selection in the playlist, andsecond, the user must click on the desired playback position on the playback
position slider located elsewhere. In Musicream sort mode, in contrast, simply
moving the mouse pointer during music playback over other discs in the series
automatically starts and stops playing those musical pieces: there is no needfor a new playback-start operation as long as the user stays within the same
playlist. In addition, clicking on the playback position slider on the periphery ofa disc right after moving the focus enables a series of operations — musical pieceselection without click and playback position speciﬁcation with a single click —to be performed simultaneously making for the smoothest zapping ever seen.
Similarity-based sticking enables a user to pick out discs from disc streams
as desired, create one playlist after another, and leave them on the Musicream
screen. Faced with multiple playlists created in this way, a user might thenthink about the best playback order for the playlists. This can be set using themeta-playlist function described next.
2.3 Meta-Playlist Function
Similar to rearranging a group of CDs on a real-world desktop, this function
enables playlists (multiple series of discs) situated on the screen to be rearranged
and their order of playback speciﬁed. This is done by treating the entire Musicream screen as a meta-playlist and playing back the playlists on the screen inorder from top to bottom.
The playback pointer for the entire screen used by this function is simply
a straight horizontal line named the “playback bar” as shown in Fig. 8. The

IPSJ Journal

Vol. 50

No. 122923–2936 (Dec. 2009)

Fig. 8 Playback rule of meta-playlist function: The playback bar falls from above and plays
playlists in order of contact.

handle at the left of the playback bar can be used to drag the bar as desired(i.e., upward or downward). Once dragged to a new position, the playback bar
starts to drop (automatically moving to the bottom of the screen) and plays backthe series of musical pieces in any maintenance-mode or sort-mode playlist (discseries) with which it comes into contact. A playlist in compact mode is ignoredin this playback.
For this function, only positional relationships in the screen’s vertical directionaﬀect playback order. Horizontal relationships have no eﬀect on playback order,
and the user can make good use of this characteristic to achieve ﬂexible playback
control. For example, to consider the best playback order of several playlists
through trial and error, the user can arrange those playlists horizontally andthen raise or lower each playlist slightly to change their vertical relationship andhear the resulting playback order as the playback bar falls. In addition, playlists
that the user does not presently wish to hear can be placed in compact modeand simply left on the screen — there is no need to delete them to prevent theminterfering with playback.
By allowing free insertion, removal, and rearrangement of groups of musicalpieces (playlists) to create meta-playlists, the meta-playlist function enables ﬂexible playlist editing. If this type of switching in units of groups were attemptedon an existing music player, the operations of selecting and inserting multiple
musical pieces would have to be repeated any number of times while remembering group boundaries. Musicream allows groups of musical pieces to be arranged

c 2009 Information Processing Society of Japan


2929

Musicream: Integrated Music-Listening Interface for Active, Flexible, and Unexpected Encounters with Musical Pieces

Fig. 9 Musicream screen snapshot with the automatic playback switch on: Enabling the automatic playback switch and letting the playback bar fall into the switch activates
automatic playback.

on the screen as individual small playlists and allows the user to change their
playback order while preserving their positional relationship in the horizontal
direction. This makes intuitive trial-and-error processing possible while using
positional memory. This is especially useful for making a long playlist by locallyarranging the order within smaller playlists and then globally arranging the orderof those smaller playlists. Playlist editing in this way is not just a method forchanging playback order. It can also be viewed as a creative way of enjoying
music with the user becoming actively involved 1 .
To facilitate the use of Musicream for playing background music, we also prepared an automatic playback mode that enables listening to musical pieces insuccession without requiring user interaction. This mode is entered by turningon the automatic playback switch and letting the playback slider fall into theswitch as shown in Fig. 9. When playback of the current musical piece ends inthis mode, Musicream starts to automatically select and play back the musicalpiece most similar in mood to that piece from among the streaming discs on the1 The importance of playlists can be understood from the way that artists carefully determine
song order in their albums and from the attention given to WWW sites that present personal
playlists (e.g., http://www.artofthemix.org/).

IPSJ Journal

Vol. 50

No. 122923–2936 (Dec. 2009)

screen (not from the playlists on the screen).
Although existing music players include a random shuﬄe playback function, an
abrupt change in mood can easily occur from one song to the next, such as whena wild rock tune follows a romantic ballad. In the automatic playback mode ofMusicream, on the other hand, the user is always listening to a new order ofmusical pieces while continuing to listen to pieces with a similar feel. Since thenext musical piece is selected from the screen at the end of the playback of thecurrent one, though, it is diﬃcult for the user to predict which piece will playnext.
For a user who encounters new music using the music-disc streaming andsimilarity-based sticking functions, and who creates playlists by trial and error
using the meta-playlist function, the act of listening to music through Musicreamis likely to become an everyday activity, in which case musical pieces played onMusicream will approximate that user’s listening experience. Accordingly, whena user would like to know what music he or she has listened to in the past,
the user should be able to ﬁnd out by checking what music has been played onMusicream. The time-machine function described next makes this possible.
2.4 Time-Machine Function
This function records all operations performed by the user on Musicream aswell as all screen changes so that the user can browse this record of the past andreturn to a desired point. This enables the user to reproduce a past Musicream
screen (i.e., recall a past listening state) and continue with operations from thatscreen as if traveling back in time with a time machine. It is also possible to copy
and paste a playlist from the past screen onto the present screen.
Reproducing a music playback order from the past in this way is diﬃcult toachieve on existing music players. It would require that playlists be intentionally
saved for future reference, but this method would only be practical if the user
knew exactly what playlists would be important in the future. The time-machinefunction works as automatic saving or loading functions that do not require
explicit commands from the user. In addition, Musicream allows the user tolabel the current point in time with a keyword or sentence that can then be used
to later recall that point.
When the button to use the time-machine function is pushed, two time-travelc 2009 Information Processing Society of Japan


2930

Musicream: Integrated Music-Listening Interface for Active, Flexible, and Unexpected Encounters with Musical Pieces

Fig. 10 Musicream screen snapshot of the time-machine function: The time-travel sliders
consist of a rewind slider (left) for travel back in time in units of seconds relative tothe present and a date/time slider (right) for specifying particular dates and times
to return to. The results of playback history search and past similarity search aredisplayed when a disc is clicked.

sliders are displayed as shown in Fig. 10.
• Rewind slider
Moving this slider takes Musicream back through time in units of seconds
relative to the present. The user can browse the operation history as if he or
she was rewinding recorded (video-taped) screen images.
• Date/time slider
Moving this slider speciﬁes a particular date and time (year, month, day,hour, minute) in the past. The slider is colored only for the intervals during
which Musicream was operating to make it easier to ﬁnd and return to a date
and time during which Musicream was being used.
The time-machine function makes it even easier to listen to music in the present
after deciding on a playback order through trial and error. Because past operations can be recovered at any time, the user need not hesitate to throw away a
musical piece picked out through the music-disc streaming function when thinking there might be a better piece available. While it is diﬃcult to intentionallyreencounter musical pieces from the past as they stream and fall from the taps

IPSJ Journal

Vol. 50

No. 122923–2936 (Dec. 2009)

in the present, it is easy to return to a past encounter and pick them up there.
We also provide two functions to support the recalling of the past through thetime-machine function. Even if speciﬁc information, such as date/time, title,
etc., corresponding to a past Musicream screen cannot be remembered, thesefunctions enable searching to be done while recalling musical pieces listened toin the past and their screens.
• Playback history searchStarting with a certain musical piece, this function searches for past screens
where this piece was listened to.
• Past similarity search
From among all musical pieces listened to in the past, this function searches
for past screens where musical pieces similar in mood to the starting musicalpiece were listened to.
The results of both of these functions are simultaneously displayed by clicking a
musical piece in the time machine operation as shown in Fig. 10.
3. Implementation
Implementing Musicream requires preprocessing of each musical piece in a music collection and implementation of the Musicream interface which is executedwith sound ﬁles of musical pieces in the MPEG Audio Layer 3 (MP3) format anda music-catalog ﬁle including the results of preprocessing. Although our current
implementation supports only MP3 sound ﬁles stored on a local hard disk, Musicream can be easily applied to online musical pieces and any music subscription
services.
3.1 Preprocessing
A music-catalog ﬁle in XML format is generated from MP3 sound ﬁles in amusic collection. The ﬁle includes title, artist name, name of the MP3 sound ﬁle,feature vector for computing music similarity, and associated disc color for eachmusical piece. The feature vector is extracted from each musical piece and thedisc color (hue and saturation) is assigned on the basis of the feature vector.
Although we can use any feature vector designed for computing music similarity, in our current implementation we use a 30-dimensional feature vector(described in Ref. 9)) obtained by automatically analyzing the mood of each mu-

c 2009 Information Processing Society of Japan


2931

Musicream: Integrated Music-Listening Interface for Active, Flexible, and Unexpected Encounters with Musical Pieces

sical piece. The eﬀectiveness of this feature vector for genre classiﬁcation has
been conﬁrmed 9) . It consists of the mean and variance of local spectral features(centroid, rolloﬀ, ﬂux, and zero-crossings) across the entire musical piece (eight
dimensions); average values of mel-frequency cepstral coeﬃcients (MFCC) acrossthe entire musical piece (ten dimensions); portion of the musical piece occupied by
low-energy intervals (one dimension); pitch content features reﬂecting periodicity
in pitch (ﬁve dimensions); and rhythmic content features reﬂecting periodicity
in beat (six dimensions). The feature vectors for all sound ﬁles are extractedusing Tzanetakis’s MARSYAS 18) , a software framework for audio analysis andsynthesis.
The disc color is determined from the color circle whose circumference andradius correspond to hue and saturation, respectively. Each musical piece is
projected into the circle according to its feature vector. Principal component
analysis (PCA) is used to reduce the dimensionality of feature vectors to a twodimensional vector on a plane. The planar coordinates consisting of the ﬁrst andsecond principal components are converted to polar coordinates, and then angle
θ is assigned to hue and distance r from the origin is assigned to saturation.
3.2 Implementation of the Musicream Interface
The Musicream interface is implemented using Macromedia Flash MX Professional 2004. It takes the music-catalog ﬁle as input and provides the fourfunctions of Musicream.
First, for the music-disc streaming function, the hue angle θ is divided into
three equal 120◦ sections, and each section is assigned to one of the three musicsupply taps. Any one tap can release only discs having hue values in the range
assigned to that tap. The number of taps can be changed by assigning angle θin diﬀerent ways. The interval of disc release can be adjusted by a slider control
in a range from 3 to 10 seconds, which is determined so that discs do not overlap
when streaming discs are expanded.
Next, for the similarity-based sticking function, the ease-of-sticking between
two discs is determined on the basis of the hue angle θ of each disc. If thediﬀerence between θ of one disc and θ of the other lies in the range from 0 to 30◦ ,the two discs will stick on the ﬁrst try. For diﬀerences of 30 to 80◦ , 80 to 130◦ ,and 130◦ and greater, the two discs will stick on the second, third, and fourth

IPSJ Journal

Vol. 50

No. 122923–2936 (Dec. 2009)

tries, respectively.
For playlist sort mode, in which discs in a series are sorted in order of similarityand hung from the base disc, the degree of similarity between two discs is deﬁned
in terms of the cosine angle (scalar product) between the 30-dimensional feature
vectors of those discs.
Finally, the time-machine function is achieved by continuously recording at
one-second intervals the snapshot information needed to reproduce past states(all discs on the screen, each tap’s internal state, date and time, etc.). Though
it depends on the number of discs on the screen, a memory capacity of about
10 Mbytes is needed to save this information in XML format for one hour ofMusicream use; there is a lot of room here, however, for compression.
4. Experimental Results
We operated Musicream using a music collection that included all 315 musicalpieces of the RWC Music Database 19) 1 and 1572 popular songs which appearedon Japanese hit charts from 2000 to 2004. We found that the proposed interfacefunctioned eﬀectively and that the four Musicream functions working in combination provided an active music-listening experience that was unique. Whenﬁrst using Musicream, users typically tried the similarity-based sticking function
after ﬁnding a musical piece that they liked in the stream of discs. After becoming more familiar with Musicream, though, instead of ﬁrst determining whether
they liked each musical piece they picked out, users went directly to sticking
many other musical pieces to the piece they picked out and then entered sortmode where they would listen to those pieces through “zapping” and then decide
which ones they liked. This was because they found that even if they did not likethe ﬁrst piece selected, they could still unexpectedly come across one that they
did like through sticking. Furthermore, by listening to a group of musical pieces
obtained through sticking, users could understand what kind of music they might
like to hear at that time.

1 Note that Musicream can support operations using a much greater number of musicalpieces. The Musicream concept should prove highly suitable for unlimited music-listening
environments, like ﬂat-rate music subscription services, as well as personal music collections.

c 2009 Information Processing Society of Japan


2932

Musicream: Integrated Music-Listening Interface for Active, Flexible, and Unexpected Encounters with Musical Pieces
Table 1 Questionnaire results with regard to the four functions: 27 subjects answered eight questions using a scale from “−3” (worst) to “+3” (best).
Subjective evaluation score
1-1. Music-disc streaming function is interesting
1-2. Music-disc streaming function is an eﬀective
way to encounter unfamiliar musical pieces2-1. Similarity-based sticking function is interesting
2-2. Want to use similarity-based sticking function
3-1. Meta-playlist function is interesting
3-2. Meta-playlist function is convenient4-1. Time-machine function is interesting
4-2. Time-machine function is convenient

−3
0
1

−2
1
0

0
0
1
0
1
0

0
0
0
2
0
0

Number of subjects
−1
0
+1
+2
2
1
2
5
0
1
4
7
0
3
0
1
0
0

2
4
4
3
2
3

2
6
4
11
2
5

8
6
4
3
8
6

+3
16
14
15
8
14
7
14
13

Table 2 Comparison between Musicream and an ordinary music player with regard to the “zapping” operation: 27 subjects
answered two questions using a scale from “−3” (ordinary music player) to “+3” (Musicream).
Subjective evaluation score
1. Which makes it easier to perform a “zapping” operation?
2. Which is more convenient for performing a “zapping” operation?

4.1 User Study of Musicream
To further analyze the advantages of Musicream, we conducted a user study
with 27 subjects (16 male, 11 female) who were not familiar with the musiccollection used here. To this end, we used musical pieces only from the RWCMusic Database in this experiment. To evaluate their subjective assessment ofMusicream after the subjects gained a good command of Musicream operation,
each subject was asked to complete a subjective questionnaire after freely using
Musicream for ﬁve minutes, excluding the time during which they received brief
instructions. As shown in Table 1, questionnaire results with regard to thefour functions indicated that more than 81.5% of the subjects rated each of
the four functions as interesting (in particular, 92.6% found the similarity-based
sticking function interesting), and that 92.6% of the subjects thought the musicdisc streaming function was an eﬀective way to encounter unfamiliar musicalpieces such as those available through ﬂat-rate, unlimited music subscription
services. In addition, another questionnaire result indicated that 96.3% of thesubjects wanted to use Musicream in the future. We also found that Musicreamwas easy enough to use without much training.

IPSJ Journal

Vol. 50

No. 122923–2936 (Dec. 2009)

−3
1
1

−2
3
0

Number of subjects
−1 0
+1
+2
6
3
2
5
6
3
4
7

+3
7
6

We then had each subject use both Musicream and an ordinary music player
with conventional playback buttons and a playlist editor 1 for three tasks —listening through the “zapping” operation, editing playlists, and ﬁnding music —and complete a subjective questionnaire comparing their operation. As shown in
Table 2, the number of subjects who rated the zapping operation on Musicream
as easier to use and more convenient exceeded the number who preferred these
operations on the ordinary player. With regard to the meta-playlist function(Table 3), 96.3% of the subjects thought that the Musicream function was easier
to use and more convenient than conventional playlist editing when the playbackorder was rearranged several times in units of groups of musical pieces. For
browsing unfamiliar musical pieces to ﬁnd a musical piece having a certain mood(e.g., uplifting music), the number of subjects who thought Musicream was moreconvenient, made music easier to ﬁnd, and was more enjoyable to use than an
ordinary music player was, respectively, 3.6, 3.8, and 5.75 times the number who
1 The ordinary music player had buttons for start, stop, pause, fast-forward, and rewind of
a musical piece, a playback slider for changing the playback position, a display of elapsed
time, and a playlist editor where musical pieces can be inserted, deleted, and reordered.

c 2009 Information Processing Society of Japan


2933

Musicream: Integrated Music-Listening Interface for Active, Flexible, and Unexpected Encounters with Musical Pieces
Table 3 Comparison between Musicream and an ordinary music player with regard to the meta-playlist function: 27 subjects
answered two questions using a scale from “−3” (ordinary music player) to “+3” (Musicream).
Subjective evaluation score
1. Which is easier to use to rearrange the playback order?
2. Which is more convenient for rearranging the playback order?

−3
0
0

−2
0
0

Number of subjects
−1
0
+1 +2
0
1
0
1
0
1
0
2

+3
25
24

Table 4 Comparison between Musicream and an ordinary music player when browsing unfamiliar musical pieces to ﬁnd a musicalpiece having a certain mood: 27 subjects answered three questions using a scale from “−3” (ordinary music player) to“+3” (Musicream).
Subjective evaluation score
1. Which is more convenient for ﬁndinga piece having a certain mood?
2. Which makes it easier to ﬁnd a piece having a certain mood?
3. Which is more enjoyable to use?

rated the ordinary music player more highly (Table 4).
These results showed that Musicream was an eﬀective way, through its four convenient functions, to enable users to browse music collections to ﬁnd unfamiliar
but interesting musical pieces.
4.2 Experiences with Musicream
One of the authors used Musicream on a trial basis for several weeks in 2005.
For this trial, 1572 popular songs which appeared on Japanese hit charts from2000 to 2004 were used. Musicream was executed on a tablet PC (Microsoft
Windows XP Tablet PC Edition) equipped with a touch screen and a digital pen
without any keyboard or mouse. All operations were therefore done using thedigital pen.
All the functions of Musicream were used in general, especially at the beginningof the trial, though the music-disc streaming function and the similarity-based
sticking function were used most often. When the main purpose was music
listening, these functions were particularly useful since they made it easy to pay
enough attention to operations and interactions.
When there were other purposes or tasks, though, and music playback wasjust for background music, such active interactions were sometimes too much.
In such cases, or when the user got tired of the interactions, the automatic

IPSJ Journal

Vol. 50

No. 122923–2936 (Dec. 2009)

−3
3

−2
1

3
2

2
0

Number of subjects
−1
0
+1 +2
1
4
4
10
0
2

3
0

5
1

8
7

+3
4
6
15

playback mode described in Section 2.3 was useful. This mode does not require
any interaction and could be used seamlessly: the mode could be turned on
while playing back the current piece, and when an interesting piece was foundduring the automatic playback, it could be taken and dragged so that other active
operations could follow. By changing the slider of music-supply taps during theautomatic playback, we could control the possible colors (moods) of played-backdiscs because this changed the streaming discs on the screen and the next piecein the automatic playback was selected from those discs.
Some drawbacks of Musicream were also found. Because the main focus ofMusicream was on operations at the level of musical pieces, operations at thelevel of artists were not supported at all although the user sometimes anticipatedtheir availability. In addition, there were no global search functions, which theuser anticipated using to ﬁnd particular musical pieces or artists.
In conclusion, the present version of Musicream is not yet perfect, though we
believe it opens up new music interface possibilities. A possible future approachis to integrate Musicream with conventional text-based music playback interfacesthat show lists of bibliographic information since Musicream is complementary
to such conventional interfaces.

c 2009 Information Processing Society of Japan


2934

Musicream: Integrated Music-Listening Interface for Active, Flexible, and Unexpected Encounters with Musical Pieces

5. Related Research
To go beyond conventional interfaces based on bibliographic information orquery-by-example retrieval, several interesting approaches for browsing musiccollections have been reported.
For example, Tzanetakis, et al. 20) developed the “GenreSpace” interface forbrowsing music collections in a three-dimensional space into which musical pieces
are projected according to their similarity; they also developed the “GenreGram”
tool for displaying, along with real-time audio input, several up-and-down cylinders corresponding to diﬀerent genres. Tzanetakis, et al. 21) have also developed
other interfaces, such as “Sound Sliders” which provides continuous aural feedback of retrieved pieces while a user moves sliders of music properties such as
tempo and beat strength.
With an emphasis on visualization on the level of musical pieces, Pampalk,
et al. 22) reported the “Islands of Music” interface that features self-organizing
maps (SOMs) and projects musical pieces onto a plane. They used a metaphorof “islands” that represent self-organized clusters of similar pieces. Using a small
focused collection of music by a single composer, it is possible to map musicalpieces in the shape of the silhouette of its composer 23) . Another visualization
technique “U-Map” 24) using a variant of SOM called Emergent SOM (ESOM)
was also proposed. The Traveller’s Sound Player 25) uses the Traveling Salesman algorithm to map musical pieces of a collection onto a circle and visualizes
the distribution of metadata (e.g., genre and tempo) having a certain value bychanging the color of the corresponding region on the circle.
As for visualization on the level of artists, Van Gulik, et al. 26) reported the“Artist Map” interface with the focus on artists and small devices. It enables
users to explore and discover music collections on small devices by projecting
artists into a two-dimensional space. Artists are drawn as dots in the space sothat similar artists are placed close together on the basis of a modiﬁed springembedder algorithm. This visualization can also be used to make playlists bydrawing paths and specifying regions on top of the visualization 27) .
By using metadata of sound ﬁles without analyzing audio signals, Torrens,
et al. 28) developed visualization techniques where musical pieces are placed in a

IPSJ Journal

Vol. 50

No. 122923–2936 (Dec. 2009)

circle, rectangle, or tree-map. When visual information related to musical pieces
is available, a “collaging” technique proposed by Bainbridge, et al. 29) is also aneﬀective way to provide leisurely, undirected interaction with a music collection.
In Musicream, visual information like jacket covers can also be used eﬀectively bydisplaying a jacket cover image on or near each music disc when such information
is available.
Although most of the above approaches share the same goal of enabling a
non-speciﬁc music search to satisfy user desires like “I want to hear something
diﬀerent that suits my taste”, Musicream is the ﬁrst interface that supports thefour functions described in Section 2. In particular, the time-machine functionis unique and has great potential. While the concept of time-machine computing
itself was proposed for a computer desktop 30) , our research is the ﬁrst to discuss
time-machine computing for music. In Musicream, the ability to reproduce all
operations from the past produces an even greater time-travel eﬀect. For example, when a user returns to the past and sticks a series of similar musical pieces
that were not previously selected (thereby creating a new future), the musicalpieces streaming on the screen revert to those that were streaming at that time.
This makes it possible to unexpectedly encounter music that was popular at thatpoint in the past.
6. Conclusion
We have described a rich, integrated music-listening interface called Musicreamthat enables seamless operation of four novel functions: “music-disc streaming,”“similarity-based sticking,” “meta-playlist,” and “time-machine.” The main contribution of Musicream is to provide a novel music-listening environment thatenables a user to interact with a huge music collection in active, ﬂexible, andcreative ways, which go beyond traditional techniques of music information retrieval.
Although the basic concept of Musicream has great potential, we have not yet
fully exploited it. In the future, for example, we could provide various speciallydesigned music taps such as a tap streaming musical pieces by a speciﬁc artist(e.g., an artist name could be typed onto the tap), a tap streaming music froma hit chart at a certain time in the past or at present, or a tap streaming musicc 2009 Information Processing Society of Japan


2935

Musicream: Integrated Music-Listening Interface for Active, Flexible, and Unexpected Encounters with Musical Pieces

selected by a famous artist or celebrity. It is also interesting to share the screen(i.e., the playlist) of Musicream with friends or anonymous users via the Internet.
If they use the same ﬂat-rate music subscription service, other users can freely
listen to those pieces as well as see the titles on the screen. For the “zapping”
operation, it will be useful to color chorus (“hook”) sections in the playback
position slider (at the disc periphery) as is done in SmartMusicKIOSK 31) : thesechorus sections can be automatically detected through the RefraiD method.
Note that the concept of Musicream is also independent of similarity measuresand dimensionality reduction techniques. We plan to use another similarity measure based on content-based or collaborative ﬁltering and another dimensionality
reduction technique such as the self-organizing map (SOM). Since Musicream
is complementary to conventional music information retrieval techniques, theirseamless integration should be the next goal. Future work will also include applying Musicream as an interface for commercial online music subscription services.
References
1) Shardanand, U. and Maes, P.: Social Information Filtering: Algorithms for Automating “Word of Mouth”, Proc. CHI ’95, pp.210–217 (1995).
2) Cohen, W.W. and Fan, W.: Web-Collaborative Filtering: Recommending Musicby Crawling The Web, Proc. WWW9 (2000).
3) Uitdenbogerd, A. and van Schyndel, R.: A Review of Factors Aﬀecting Music
Recommender Success, Proc. ISMIR 2002, pp.204–208 (2002).
4) Alghoniemy, M. and Tewﬁk, A.H.: A Network Flow Model for Playlist Generation,
Proc. ICME 2001 (2001).
5) Pauws, S. and Eggen, B.: PATS: Realization and User Evaluation of an Automatic
Playlist Generator, Proc. ISMIR 2002, pp.222–230 (2002).
6) Logan, B.: Content-Based Playlist Generation: Exploratory Experiments, Proc.
ISMIR 2002, pp.295–296 (2002).
7) Aucouturier, J.-J. and Pachet, F.: Scaling Up Music Playlist Generation, Proc.
ICME 2002 (2002).
8) Logan, B.: Music Recommendation from Song Sets, Proc. ISMIR 2004, pp.425–428(2004).
9) Tzanetakis, G. and Cook, P.: Musical Genre Classiﬁcation of Audio Signals, IEEE
Trans. Speech and Audio Proc., Vol.10, No.5, pp.293–302 (2002).
10) Aucouturier, J.-J. and Pachet, F.: Music Similarity Measures: What’s the Use?,
Proc. ISMIR 2002, pp.157–163 (2002).
11) Paulus, J. and Klapuri, A.: Measuring the Similarity of Rhythmic Patterns, Proc.

IPSJ Journal

Vol. 50

No. 122923–2936 (Dec. 2009)

ISMIR 2002, pp.150–156 (2002).
12) Foote, J., Cooper, M. and Nam, U.: Audio Retrieval by Rhythmic Similarity, Proc.
ISMIR 2002, pp.265–266 (2002).
13) Dixon, S., Pampalk, E. and Widmer, G.: Classiﬁcation of Dance Music by Periodicity Patterns, Proc. ISMIR 2003, pp.159–165 (2003).
14) McKinney, M.F. and Breebaart, J.: Features for Audio and Music Classiﬁcation,
Proc. ISMIR 2003, pp.151–158 (2003).
15) Tsai, W.-H., Wang, H.-M., Rodgers, D., Cheng, S.-S. and Yu, H.-M.: Blind Clustering of Popular Music Recordings Based on Singer Voice Characteristics, Proc.
ISMIR 2003, pp.167–173 (2003).
16) Pampalk, E.: A MATLAB Toolbox to Compute Music Similarity from Audio,
Proc. ISMIR 2004, pp.254–257 (2004).
17) Huang, H.-H., Nomura, S., Sumi, Y. and Nishida, T.: Toward Contents Management with Somatic Sensation, Proc. 19th Annual Conference of the JapaneseSociety for Artificial Intelligence, pp.1D2–02 (2005). (in Japanese).
18) Tzanetakis, G. and Cook, P.: MARSYAS: A Framework for Audio Analysis, Organized Sound, Vol.4, No.3, pp.169–175 (2000).
19) Goto, M.: Development of the RWC Music Database, Proc. ICA 2004, pp.I–553–
556 (2004).
20) Tzanetakis, G., Essl, G. and Cook, P.: Automatic Musical Genre Classiﬁcation of
Audio Signals, Proc. ISMIR 2001, pp.205–210 (2001).
21) Tzanetakis, G., Ermolinskyi, A. and Cook, P.: Beyond the Query-by-Example
Paradigm: New Query Interfaces for Music Information Retrieval, Proc. ICMC
2002, pp.177–183 (2002).
22) Pampalk, E., Dixon, S. and Widmer, G.: Exploring Music Collections by Browsing
Diﬀerent Views, Proc. ISMIR 2003, pp.201–208 (2003).
23) Mayer, R., Lidy, T. and Rauber, A.: The Map of Mozart, Proc. ISMIR 2006 (2006).
24) Mörchen, F., Ultsch, A., Nöcker, M. and Stamm, C.: Databionic visualization ofmusic collections according to perceptual distance, Proc. ISMIR 2005, pp.396–403(2005).
25) Schedl, M., Pohle, T., Knees, P. and Widmer, G.: Assigning and Visualizing Music
Genres by Web-based Co-Occurrence Analysis, Proc. ISMIR 2006 (2006).
26) van Gulik, R., Vignoli, F. and van de Wetering, H.: Mapping Music in the Palm ofYour Hand, Explore and Discover Your Collection, Proc. ISMIR 2004, pp.409–414(2004).
27) van Gulik, R. and Vignoli, F.: Visual playlist generation on the artist map, Proc.
ISMIR 2005, pp.520–523 (2005).
28) Torrens, M., Hertzog, P. and Arcos, J.-L.: Visualizing and Exploring Personal
Music Libraries, Proc. ISMIR 2004, pp.421–424 (2004).
29) Bainbridge, D., Cunningham, S.J. and Downie, J.S.: Visual Collaging of Music in
a Digital Library, Proc. ISMIR 2004, pp.397–402 (2004).

c 2009 Information Processing Society of Japan


2936

Musicream: Integrated Music-Listening Interface for Active, Flexible, and Unexpected Encounters with Musical Pieces30) Rekimoto, J.: Time-Machine Computing: A Time-centric Approach for the Information Environment, Proc. UIST ’99, pp.45–54 (1999).
31) Goto, M.: A Chorus-Section Detection Method for Musical Audio Signals and
Its Application to a Music Listening Station, IEEE Trans. on Audio, Speech andLanguage Processing, Vol.14, No.5, pp.1783–1794 (2006).

(Received March 18, 2009)(Accepted September 11, 2009)(Original version of this article can be found in the Journal of Information Processing Vol.17, pp.292–305.)

Takayuki Goto received the Master’s degree at Graduate
School of Media and Governance, Keio University, Japan, in 2007.
He is currently a Ph.D. student in the Department of Informatics,
School of Multidisciplinary Science, The Graduate University for
Advanced Studies (Sokendai). His research interests include information retrieval and human-computer interaction. He received
2 awards, including the IPSJ Interaction 2005 Best Interactive
Presentation Award.

Masataka Goto received the Doctor of Engineering degreefrom Waseda University, Japan, in 1998. He is currently a Leaderof Media Interaction Group, Information Technology Research Institute at the National Institute of Advanced Industrial Scienceand Technology (AIST). He serves concurrently as a Visiting Professor in the Department of Statistical Modeling, The Institute of
Statistical Mathematics, and an Associate Professor (Cooperative
Graduate School Program) in the Department of Intelligent Interaction Technologies, Graduate School of Systems and Information Engineering, Universityof Tsukuba. He received 24 awards, including the Commendation for Scienceand Technology by the Minister of MEXT “Young Scientists’ Prize”, DoCoMo
Mobile Science Awards “Excellence Award in Fundamental Science”, IPSJ Nagao
Special Researcher Award, IPSJ Best Paper Award, IPSJ Yamashita SIG Research Awards, and Interaction 2003 Best Paper Award.

IPSJ Journal

Vol. 50

No. 122923–2936 (Dec. 2009)

c 2009 Information Processing Society of Japan