IUI 2016 • IUI for Entertainment and Health

March 7–10, 2016, Sonoma, CA, USA

PlaylistPlayer: An Interface Using Multiple Criteria to
Change the Playback Order of a Music Playlist
Tomoyasu Nakano
Masahiro Hamasaki
Jun Kato
Masataka Goto
National Institute of Advanced Industrial Science and Technology (AIST), Japan{t.nakano, jun.kato, masahiro.hamasaki, m.goto}@aist.go.jp
 Unsorted (Original)  Ascending order  Descending order  Random order  Smooth order

ABSTRACT

We propose a novel interface that allows the user to interactively change the playback order of multiple songs by choosing one or more criteria. The criteria include not only thesong’s title and artist name but also its content automaticallyestimated by music/singing signal processing and artist-levelsocial analysis. The artist-level social information is discovered from Wikipedia and DBpedia. With regard to manipulating playback order, existing interfaces typically allow theuser to change it manually or automatically by choosing oneof a few types of criteria. The proposed interface, on the other
hand, deals with nine properties and multiple integrations ofthem (e.g., vocal gender and beats per minute). To realize theordering by multiple criteria, a distance matrix is computedfrom the criteria vectors and is then used to estimate paths forascending, descending, and random orders by applying principle component analysis or to estimate a path for a smoothorder by solving the travelling salesman problem.

Metadata

Audio-based
information

Relationships among songs○␣ the currently played back song and○ the others

Button to start
playlist playback

Figure 1. An example PlaylistPlayer screen.

interface (Figure 1). The following three kinds of properties
can be visualized and used to change the playback order:

ACM Classiﬁcation Keywords(type 1: absolute values) meta-data:
intrinsic information (song length) and information given
by a listener’s behavior (the number of views),(type 2: absolute values) audio-based information:
vocal information (vocal gender) and musical information(beats per minute: BPM),(type 3: relative values) relationships among songs:
musical commonness [14] (a song’s similarity to a set ofsongs) of vocal timbre, musical timbre, and rhythm; andsocial commonness (artist-level social relationships between a song and a set of songs) of artist background information on the web.

H.5.5. Information Interfaces and Presentation (e.g., HCI):Sound and Music Computing – Signal analysis, synthesis,
and processing; H.5.2. Information Interfaces and Presentation (e.g., HCI): User Interfaces – GUI.
Author Keywords

Music playlist; playback order; visualization; vocal gender;
musical commonness; social commonness.
INTRODUCTION

Playlists, lists of songs played back automatically, are a popular way of listening to music. The playback order can be
determined using metadata such as artist names and song titles, and the “freshness” of a playlist listened to repeatedly
can be increased by randomly shufﬂing the playback order.

Any of those properties can be used as a re-ordering criterion, and two or more properties (e.g., vocal gender, BPM
and musical commonness) can be used as re-ordering criteria.
Integrating multiple properties enables designing a variety of
playback orders that match the user’s needs.

We propose a playlist playback interface, PlaylistPlayer, thatenriches the listener’s experience by allowing interactively
changing the playback order based on various properties ofthe songs and their artists. These properties, derived fromcontent and from web, are visualized in the extended playlist

RELATED WORK

Previous work on playlist generation is related to our interface’s changing playback order, and previous work on playlist
visualization in order to discover a song is related to our interface’s visualization.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor proﬁt or commercial advantage and that copies bear this notice and the full citationon the ﬁrst page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior speciﬁc permission
and/or a fee. Request permissions from Permissions@acm.org.
IUI’16, March 07–10, 2016, Sonoma, CA, USA
Copyright is held by the owner/author(s). Publication rights licensed to ACM.
ACM 978-1-4503-4137-0/16/03...$15.00.
DOI: http://dx.doi.org/10.1145/2856767.2856809

Playlist Generation

Common approaches to creating playlists on existing music
players such as iTunes are to manually select songs, using
some conditions to create complex playlists (Smart Playlist),or to select songs automatically (Genius).

186

IUI 2016 • IUI for Entertainment and HealthSong title

Artist nameMarch 7–10, 2016, Sonoma, CA, USA(Type 1) Song Number
duration of plays

Female singer
Male singer
Female singer
Figure 3. The random order of the vocal gender.
(Type 2)Vocal
gender

BPM(Type 3)

Semantic commonness
Musical commonness
Wikipedia
vocal
timbre rhythm Category / Related entry

1 property

2 properties

3 properties

top songbottom song

Figure 2. The tabular visualization of all properties.

Figure 4. Re-ordering by the smooth order using multiple properties.

On the other hand, much research has also been focused on
playlist generation [2, 18]. Research on generating playlists
based on smooth transition between one song and the previous and next songs (acoustically similar pieces) [4, 17] is related to our system in terms of playback order. And research
considering human behavior during playback (e.g., skippinga song) and generating a playlist by dynamically adapting to
the user’s preferences [11,15] is related to our system in terms
of using human behavior (i.e., the number of views).

rhythm. Second, we crawl the web for artist-level social relationships among all songs in the playlist.

PlaylistPlayer, however, uses metadata, audio data, and behavioral data not for ‘generating a new playlist but for changing the order in which an existing playlist is played back’.
Moreover, multiple criteria can be used by integrating the individual criteria.

PlaylistPlayer can change playback order according to thethree types of properties listed in the INTRODUCTION. Figure 2 shows all three types with some tabular visualization
methods by reference to [16]. Type-1 properties are represented by blue rectangles in horizontal bar graphs. Type-2
properties are represented by colored rectangles (blue or red).
For example, if a male probability of vocal gender is high, therectangle is colored blue and placed at the left side. And type3 properties are represented by green circles whose sizes depend on relationships (musical or social commonness). The
reason for using the commonness instead of the similarity between two songs is to ﬁt to the tabular type visualization.

Finally, we implement an interface for interactions duringchanging playback order and visualization. In this section we
describe the interface and the interactions. The re-ordering,
signal processing and web mining methods are described indetail in the next section.
Change Playback Order and Visualization

Playlist Visualization

There has been much work in which visualization of relationships among songs was used to browse, to explore, andto discover a song [12]. Some research focusing on combining visualization with playlist generation is related to oursystem, such as interactive playlist generation [20], generating a thumbnail image that represents audio features of aplaylist [19], and printing out album covers for use as tangible interfaces of playlists [8].

To change the order, sort/unsorted marks, a smooth mark,and a random mark are placed on top of each property column (Fig. 1). A single triangular shape means ascending ordescending order, the circle shape means all song transition
are smooth (including a transition between top and bottom
songs), and the curved arrow means random order. Random
order means that all the songs in the playlist are ordered alternately with regard to the selected properties. For example,
if the random mark of the “Gender” property is checked (Fig.
3), a song sung by a male singer is played after a song sung by
a female singer. When there are two triangular shapes on one
property, that property is not considered when determining
the order (i.e., the order is the original predeﬁned order).

There has also been work on visualizing artist relationships based on musical background (e.g., female solo-singer),moods, tempo, genres, active years and decades [7, 20], andassociated activities (i.e., members of a band) [9].
PlaylistPlayer, in contrast, uses tabular type visualization as
an extension of the standard playlist. Our system visualizes
not only static values (i.e., metadata and acoustic characteristics) but also dynamically changing values (i.e., relationships
among songs). In addition, artist relationship includes artists’personal information on the web.

The listener can change the order by clicking the marks of
one or more properties each time. If two or more properties
are selected, these marks are coordinated (Fig. 4).

PLAYLISTPLAYER: AN EXTENDED PLAYLIST

PlaylistPlayer is a playlist interface which is an extended classical list-view style playlist. To realize PlaylistPlayer, ﬁrst themusical characteristics of the audio data for all the songs in
the playlist are estimated automatically. The characteristics
used in the current implementation are BPM, vocal gender,
commonness of vocal timbre, musical timbre, and musical

IMPLEMENTATION

To test the interface, we used 19 songs as listed in Figure
5. They were chosen among 415 English songs performedby various types of artists (solo singers, male/female singers,

187

IUI 2016 • IUI for Entertainment and Health

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

Estimated
male/female probability

Artist name / Song title
Avril Lavigne / ComplicatedBrian Mcknight / Still
Britney Spears / ...Baby One More Time
Celine Dion / My Heart Will Go On
Christina Aguilera / Beautiful
Eminem / Lose Yourself
Erykah Badu / On & On
Evanescence / Going Under
Five for Fighting / Superman (It's Not Easy)
Jamiroquai / Virtual Insanity
John Mayer / Daughters
Joss Stone / You Had Me
Madonna / Music
Michael Jackson / You Are Not Alone
Norah Jones / Don't Know Why
Paul Simon / You're The One
Queen / We Are The Champions
Sting / Fragile
Tracy Chapman / Give Me One Reason

PCA-based path

Euclidean distance matrix

Female 0.65
1
1
2
0.20
3
0.9
0.75
4
0.61
0.8
5
0.45
6
0.14
0.7
7
0.68
8
0.60
0.6
9
0.25
0.5
10
0.46
11
0.19
0.4
12
0.75
13
0.77
0.3
14
0.41
15
0.2
0.61
16
0.14
0.1
17
0.18
18
0.15
0
19
Male 0.19
song ID 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

0.8

3TSP-based path13
3song ID

0.7

Male/female probabilities

song
ID

March 7–10, 2016, Sonoma, CA, USA

13

12

3

song ID

7

12

7

1

1
15

0.6

15

4

8

4

8

0.5
10

5

10

5

0.4 14

14

0.3
9

2

0.2

11

2

9

19

17

11

6 18

0.1

60

80

2
17

100

120

19

6 18

16

16

140

BPM

BPM

BPM

140

1

100
60

Male/female probabilities

Male/female probabilities

0.8
0.4

0

Figure 6. A Euclidean distance matrix calculated from esFigure 5. Songs used in the evaluation. timated male/female probabilities in the 19 songs.

0

1

5

10

15

19

increasing / decreasing

1

5

10

smooth

15

19

order

Figure 7. Two types of paths are estimated.

bands, or groups). They were taken from commercial music
CDs (Billboard Top Rock’n’Roll Hits 1968-1972, Billboard
Top Hits 1975-1989 and GRAMMY NOMINEES 1996-2005).

Figure 7 shows that path-estimation results obtained by taking
into account two properties (vocal gender and BPM).

Re-Ordering using multiple criteria

BPM (type 2) of each song is estimated by using the medianvalue of the time intervals between individual beats during
one song. To estimate beats, we use modules of Songle [6].

Music/singing signal processing

All properties among songs can be represented as an N × N
Euclidean distance matrix (N is the number of songs of theplaylist). If the listener selects two properties as criteria forre-ordering, such as vocal gender and BPM, a distance between two songs is calculated from the 2-dimensional vector.
Before calculating the distance, each vector is normalized bysubtracting the mean and dividing by the standard deviation.
The Euclidean distance matrix A is deﬁned asD
A = (a ) 1≤i≤N , a = (x (d)2 − x (d)2 ),
(1)
ij

1≤j≤N

ij

i

Since we focus on vocal timbre, musical timbre, and rhythm,
we extract acoustic features representing them from the musical audio signals [14]. These features are used to estimate thevocal gender (type 2), and the musical commonnesses (type3) of songs.
Vocal Gender Estimation (type 2)

Our system also uses a vocal gender estimation method [10].
Male/female probabilities of each reliable frame (see [10])are estimated using a probabilistic Support Vector Machine(SVM) [3] and are reduced to a single value by taking themedian across the SVM output.

jd=1

where xi and xj are the D-dimensional normalized vectorsof two songs (see Figure 6 for concrete examples).
For changing playback order according to those properties,
we compute a one-dimensional vector from the matrix torealize the ascending/descending/random order. The onedimensional vector is computed by applying principle component analysis (PCA) for reduction of dimensions of the distance matrix. Then the one-dimensional vector is sorted.

Fifteen songs (13 Japanese and 2 English) sung by ﬁfteen
male singers and thirteen songs (11 Japanese and 2 English)sung by thirteen female singers are used as the training set ofeach model. These songs are solo vocal performances and arefrom the RWC Music Database (RWC-MDB-P-2001) [5].

The path for smooth transition is estimated by solving thetravelling salesman problem (TSP). Although the TSP wasused for getting acoustically-smoothed transition in previous
works [4, 17], the originality of the PlaylistPlayer is that the
TSP used to integrate multiple criteria (Fig. 4).

We use a musical commonness estimation method [14] basedon latent Dirichlet allocation (LDA). A song-set (playlist)
model can be obtained by summing the hyperparameters of
the topic distributions of song models [14].

Musical Commonness Estimation (type 3)

To train the LDA-based topic model, we used the 415 songs.

The PCA-based and TSP-based methods can scale up to atleast hundreds of songs (completed in a second on a personal
computer). Moreover, their computational efﬁciencies can beincreased by using the EM algorithm for PCA [1] and someapproximate algorithms for TSP [17]. For the PCA, the mostcomputationally demanding steps of the EM algorithm areO(N DM ), where M denotes the number of principle components [1]. For the TSP, the Minimum Spanning Tree isO(m log m) with m being the number of edges [17].

Web mining

To provide different type of relations among songs, we
focus on artist properties. The artist information is obtained from Wikipedia (http://www.wikipedia.org/) and DBpedia [13]. DBpedia has structured information extracted
from Wikipedia. From each artist entry (page) are extracted
phrases in the Categories section (e.g., Grammy Awardwinning artists) and related entries’ names that have a link

188

IUI 2016 • IUI for Entertainment and Health

March 7–10, 2016, Sonoma, CA, USA

experience with arrangement/composition. The playlist consisted of the 19 songs (Figure 5). All users knew more than 1
song, and four users knew more than 10 songs. After the trial
usage, we asked the users to write comments about the two
primary functions of the system: 1) song information visualization and 2) changing the playback order.

RelatedentriesCategories

from http://en.wikipedia.org/wiki/Avril_Lavigne

Positive comments about the capabilities and potential of theinterface are listed below.

Figure 8. A Wikipedia artist entry (page) and the kinds of information.

Visualization of multiple properties This is helpful when
exploring the playback order that meets my expectation.
Automatic re-ordering based on content-based criteria
1) Easy to use and much appreciated. 2) Helpful for ﬁnding new songs similar to songs of favorite artists. 3) The
smooth order creates smooth transitions between songs
which are suitable for background music. 4) The random
order gives fresh listening experience for a worn-out set ofsongs.
Both functions 1) Both were useful for listening songs in a
new perspective. 2) Both are likely to be helpful for generating a new playlist. 3) The properties for social commonness were useful because they matched the impression of
the songs/artists.

with another entry on Wikipedia. An example of the extracted
information is shown in detail in Figure 8.
We investigated social information among 335 artists of the
415 songs. The total numbers of social information are 2,154
categories and 15,658 related entries. There was musicrelated information (e.g., Musical quartets, American pop
singers, and genre), personal information (e.g., 1942 births),
and information about awards.
Social Commonness (type 3)

We estimate social commonness by using the categories andthe related entries as social annotations. The estimation procedure is similar to the musical commonness estimation procedure based on the LDA. An artist-set (playlist) model canalso be obtained by integrating artist models.

There were also negative comments pointing out the limitations of the current implementation.

Evaluation

Visualization of multiple properties 1) The visualizationof the social commonness properties was not comprehensive enough. 2) There are sometimes mismatches between
visualization results and auditory impression. 3) The type3 visualization was sometimes confusing.
Automatic re-ordering based on content-based criteria
1) Since there are a lot of properties (nine), the re-ordering
operation by integrating them is a bit more complex. 2)
The commonness order is hard to comprehend.
Both functions / Other perspective 1) There is too muchinformation for casual listening. 2) Some properties require prior knowledge of music to comprehend the ordering rules. 3) Functions to compare results of sorting and toﬁlter the results based on multiple criteria are desired. 4)
Released date of songs can be added as a new property forsorting.

Accuracies of the proposed estimation methods are calculated
for the 19 songs. To evaluate the vocal gender estimation, estimated male/female probabilities are shown in Figure 6 andthey are binarized (“male” if the probabilities are equal to or
lower than 0.5, “’female” if the probabilities are greater than0.5). The classiﬁcation rate was 94.7% (18/19 songs).
Estimated BPMs are comparable to annotated BPMs by a musician. To evaluate the estimation method, we computed ﬁve
BPMs for each song based on ﬁve beat structure candidates
estimated by Songle [6]. Then, we chose the BPM that is theclosest to the annotated BPM. A root-mean-square value ofthe difference between the chosen (estimated) BPMs of 19
songs and the annotated BPMs was 0.78.
Estimated musical commonness and social commonness areevaluated based on the Pearson product-moment correlation
coefﬁcients between estimated commonness of each song andthe number of songs having high similarity with the song[14]. If the correlation coefﬁcients are high, the estimatedcommonness is appropriate. This means a song having higher(lower) commonness and it is very similar (is not similar) tosongs of a song set. The correlation coefﬁcients of the threemusical elements (vocal timbre, musical timbre and rhythm)
were 0.87, 0.91 and 0.80, and of the two social information(category and related entry) are 0.81 and 0.85.

These comments suggest that the interface is effective, butsome visualizations/functions should be improved.
CONCLUSION

This paper presents PlaylistPlayer, a playlist playback interface that can be used to change the playback ordering of existing playlists by using multiple criteria interactively. Although
this paper did not discuss the “repeat” function, which is an
standard function to use for playlist playback, the proposed
criteria (or combinations of them) can be used to change after ﬁnishing playback of all songs to create various kinds ofrepeats. In addition, we intend in future work to further improve the interaction in ways inferred from user feedback.

USER FEEDBACK

To investigate the capabilities, limitations, and potential of
our interaction design, we asked seven users to use the systemfor 20 minutes and collected preliminary user feedback. We
chose the users (six males and one female) so that they havedifferent levels of musical experience/knowledge. Six users
had experience with musical instruments, and three users had

ACKNOWLEDGMENT

This paper utilized the RWC Music Database (Popular Music). This work was supported in part by CREST, JST.

189

IUI 2016 • IUI for Entertainment and Health

March 7–10, 2016, Sonoma, CA, USA

11. Hu, Y., and Ogihara, M. NextOne Player: A music
recommendation system based on user behavior. In
Proc. ISMIR 2011 (2011), 103–108.

REFERENCES

1. Bishop, C. M. Pattern Recognition and Machine
Learning. Springer-Verlag New York, Inc., 2006.

12. Langer, T. Music information retrieval & visualization.
In Trends in Information Visualization (2010), 15–22.

2. Bonnin, G., and Jannach, D. Automated generation ofmusic playlists: Survey and experiments. Journal of
ACM Computing Surveys (CSUR) 47, 2 (2015), 1–35.

13. Lehmann, J., Isele, R., Jakob, M., Jentzsch, A.,
Kontokostas, D., Mendes, P. N., Hellmann, S., Morsey,M., van Kleef, P., Auer, S., and Bizer, C. DBpedia - alarge-scale, multilingual knowledge base extracted from
Wikipedia. Semantic Web Journal (2012), 1–5.

3. Chang, C.-C., and Lin, C.-J. Libsvm: A library forsupport vector machines. ACM TIST 2 (2011), 1–39.
4. Flexer, A., Schnitzer, D., Gasser, M., and Widmer, G.
Playlist generation using start and end songs. In Proc.
ISMIR 2008 (2008), 173–178.

14. Nakano, T., Yoshii, K., and Goto, M. Musical similarity
and commonness estimation based on probabilistic
generative models. In Proc. ISM 2015 (2015), 197–204.

5. Goto, M., Hashiguchi, H., Nishimura, T., and Oka, R.
RWC music database: Popular, classical, and jazz music
databases. In Proc. of ISMIR 2002 (2002), 287–288.

15. Pampalk, E., Pohle, T., and Widmer, G. Dynamic
playlist generation based on skipping behavior. In Proc.
ISMIR 2005 (2005), 634–637.

6. Goto, M., Yoshii, K., Fujihara, H., Mauch, M., andNakano, T. Songle: A web service for active music
listening improved by user contributions. In Proc. ISMIR
2011 (2011), 311–316.

16. Perin, C., Dragicevic, P., and Fekete, J.-D. Revisiting
bertin matrices: New interactions for crafting tabularvisualizations. IEEE Trans. Visualization and Computer
Graphics 20, 12 (2014), 2082–2091.
17. Pohle, T., Pampalk, E., and Widmer, G. Generating
similarity-based playlists using traveling salesman
algorithms. In Proc DAFx’ 05 (2005), 1–6.

7. Gouyon, F., Cruz, N., and Sarmento, L. A Last.fm andYouTube mash-up for music browsing and playlist
edition. In Proc. ISMIR 2011 (2011), 1–2.
8. Graham, J., and Hull, J. J. iCandy: a tangible userinterface for iTunes. In Proc. CHI2008 (2008),
2343–2348.

18. Song, Y., Dixon, S., and Pearce, M. A survey of music
recommendation systems and future perspectives. In
Proc. CMMR 2012 (2012), 395–410.

9. Hamanaka, M., and Yoshiya, M. BandNavi:
Band-member backtracker based on member history
information. Journal of Communication and Computer9, 2 (2012), 114–121.

19. Uota, T., and Itoh, T. GRAPE: A gradation based
portable visual playlist. In Proc. IV 2014 (2014),
361–364.

10. Hamasaki, M., Goto, M., and Nakano, T. Songrium: A
music browsing assistance service with interactive
visualization and exploration of a web of music. In Proc.
WWW 2014 (2014), 523–528.

20. van Gulik, R., and Vignoli, F. Visual playlist generationon the artist map. In Proc. of ISMIR 2005 (2005),
520–523.

190