Songrium: A Music Browsing Assistance Service
Based on Visualization of Massive Open Collaboration
Within Music Content Creation Community
Masahiro Hamasaki

Masataka Goto

National Institute of Advanced Industrial Scienceand Technology (AIST)
1-1-1 Umezono, Tsukuba,
Ibaraki 305-8568, Japan

National Institute of Advanced Industrial Scienceand Technology (AIST)
1-1-1 Umezono, Tsukuba,
Ibaraki 305-8568, Japanmasahiro.hamasaki@aist.go.jp

m.goto@aist.go.jp

ABSTRACT

1.

This paper describes a music browsing assistance service, Songrium (http://songrium.jp), that helps a user enjoy songs while seeing visualization of open collaboration.
Songrium focuses on open collaboration for music content
creation on the most popular Japanese video-sharing service. Since this open collaboration generates more than halfa million video clips with a rich variety of music content,
we call it massive open collaboration. To develop a sharedunderstanding of this collaboration we have analyzed, we developed Songrium that visualizes relations among both original songs and derivative works generated from the collaboration. Songrium also features a social annotation framework
to verbalize and share various relations among songs, anda flexible ranking mechanism to find interesting songs. After we launched Songrium in August 2012, more than 7,000
users have used our service in which over 98,000 songs and
520,000 derivative works have automatically been registered.
We hope Songrium will not only encourage creators to create more derivative works, but also attract consumers toparticipate in the collaboration as creators.

The goal of this research is to encourage consumers andcreators to enjoy massive open collaboration for music content. Users of the most popular Japanese video-sharing
service Nico Nico Douga (http://www.nicovideo.jp) spontaneously started collaboration on creating derivative worksof original songs in the form of video clips on the service
and uploading them. Such a collaboration can enable thecreation of “derivative works of derivative works”. This collaboration is open because anyone can join without explicit
coordination or contract, though creators of derivative works
often naturally express their appreciation for the use of their
original work.
The open collaboration in this paper is similar to wellknown open collaboration [24][7] in Wikipedia or open
source software (OSS) because both collaborations are basedon spontaneous motivations and self-organizing growth.
Such existing open collaboration usually needs to merge andorganize contributions from collaborators. Given the nature of music content, however, our open collaboration is
diﬀerent. The aggregation of personal contributions (video
clips) itself is valuable and generates a rich variety of music
content. This unorganized variety further generates diverse
derivative works.
The most intensive open collaboration on the Nico Nico
Douga service emerges from music content related to thesinging synthesis technology VOCALOID [16]. It forms aninteresting music content creation community that has already generated more than half a million video clips according to our analysis. We therefore call it massive open collaboration. A reuse network of this large number of derivative
works in the VOCALOID community on Nico Nico Douga
forms a huge directed graph where each node denotes a different video clip and each edge has a direction from an original work to a derivative work [5]. Figure 1 shows a small
excerpt of the huge reuse network in the VOCALOID community. In our previous work [11, 10], we analyzed this network and realized that it was too huge for both consumers
and creators to understand the whole picture.
We therefore developed a web service called Songrium(http://songrium.jp) that helps a user browse original songs
and their derivative works by visualizing their relations (Figure 2). Original songs are visualized as if they are stars ina planetarium. Their positions are automatically arranged
so that songs with similar moods can be closely located andCategories and Subject Descriptors
H.3.5 [Online Information Services]: Web-based services; H.5.5 [Sound and Music Computing]: Signal analysis, synthesis, and processing

General Terms
DesignKeywords
Massive open collaboration, Music interface, Visualization,
Web service, User-generated content, Social tagging
Permission to make digital or hard copies of part or all of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. Copyrights for thirdparty components of this work must be honored. For all other uses, contact
the Owner/Author.
Copyright is held by the owner/author(s).
WikiSym ’13 Aug 05-07 2013, Hong Kong, China
ACM 978-1-4503-1852-5/13/08.
http://dx.doi.org/10.1145/2491055.2491059

INTRODUCTION

Original song (2,592,882 PVs)

Performing
instruments

Dancing(1,680,188 views)

D

M, B, V

Singing

D

D

M, V

M, B, P

(783,424 views)

D

M, B, P

M
M, B, PV

M, V

(302,974 views)

Performing
instruments

B, P

Arranging(12,543 views)(381,143 views)

Creating music video

Mashup(215,244 views)

Dancing(36,217 views)
V

Creating music video
with 3D character(over 3,000 video clips)

Mashup(over 2,000 video clips)(28,130 views)

3D

Creating music video
with 3D character(246,848 views)
M, V, B

B

Singing(45,469 views)

Original songs

3D

Singing in English(101,994 views)

M (Lyrics in English)

Singing

Mashup(108,456 views)(39,628 views)

B, P

Category of reused materials
M
V
B
D
3D
P

Musical composition
Vocal track
Backing track
Dance motion
3D modelVideo picture

Figure 1: Excerpt of the huge reuse network in the VOCALOID community on the Nico Nico Douga service.
Each video clip is denoted as a node (circle) with the name of its category and page views. The color ofa node corresponds to its category. Each directed edge represents derivation and is accompanied by labelsthat explain which parts of the original work are reused in its derivative work. Edges from an original song
to further derivative works of derivative works are omitted for simplicity. The root node of this excerpt isan original song having the video ID “sm12825985” on Nico Nico Douga and has generated a wide variety ofderivative works.

1) Music Star Map

2) Planet View

3) Arrow Tag

4) Filter

Figure 2: Screenshots of Songrium.

be easily listened to by the user. When the user clicks an
original song, its derivative works appear and rotate around
it as if they are planets orbiting a star (i.e., the originalsong). The user can thus find open collaboration related tothe song.
Songrium also enables users to annotate, share, and enjoy
relations between original songs. In this paper, we propose
a new social annotation framework where a user voluntarily
puts a directed tag, called an Arrow Tag, on a relation fromone song to another. Each Arrow Tag represents an implicit

or explicit relation between songs and is immediately sharedwith other users after being annotated by anyone or being
automatically generated from web information. A user canclick Arrow Tags to follow and enjoy various relations to encounter unknown songs. Furthermore, Songrium supportsa customizable ranking mechanism where a user can change
criteria to generate a personal ranking of original and derivative works to find interesting songs.
Although we launched Songrium in August 2012 and over
7,000 users have used our service, this paper is the first academic report of this service. Since more than 98,000 original
songs and 520,000 derivative works have already been registered and new songs are also automatically registered every
day, Songrium is the only large-scale web service that can
provide a comprehensive overview of massive open collaboration within the music content creation community for
VOCALOID.
This paper is organized as follows. Section 2 explains
massive open collaboration within the VOCALOID community on Nico Nico Douga. Section 3 introduces functions
provided by Songrium and Section 4 describes how they
are implemented. Section 5 presents our experiences with
Songrium and Section 6 discusses related research and future potential of Arrow Tags. Section 7 summarizes thispaper’s contributions.

2.

OPEN COLLABORATION WITHIN
THE VOCALOID COMMUNITY ON
NICO NICO DOUGAThis section first describes the Nico Nico Douga service asa platform of the open collaboration and the VOCALOID
community in which massive open collaboration emerges. It
then reveals key factors that contribute to the collaborationand discusses its open issues.

2.1 Nico Nico Douga
Nico Nico Douga is a popular video communication web
service started in December 2006 and had more than 30
million users (including more than 1.8 million paying users)
in Japan in December 2012. It was developed by Dwango
Co. Ltd. and managed by Niwango Co. Ltd. As with othervideo sharing services (e.g., YouTube [27, 31]), users canupload, share, and view video clips.
In addition, the Nico Nico Douga service supports a novel
networked text communication that enables users to post,share, and watch time-synchronous text comments (semantic information) related to video clips. Recent text comments by anonymous users are overlaid on the video screen
in synchronization with the video playback. Each comment
typed in at a specific playback time within a video clip flows
from the right to the left over the video screen at that time.
Comments can therefore be related to various events in avideo clip and can be shared with users who watch the clip
currently or in the future. Although relevant web-based
video annotation systems in which comments can be associated with events in a video clip exist [30, 15], these systems
do not support the comment overlay functionality.
Because the overlaid comments can create a sense ofshared watching/listening experience, called Pseudo Synchronized Communication by Satoshi Hamano, users can feelas though they are enjoying the experience together with acrowd of anonymous users. Users usually post impressions,interpretations, and feelings about the video/music content.
Such time-synchronous comments are helpful for creators to
get feedback from consumers (listeners) and often motivate
creators to upload original video clips much more.

2.2 Open Collaboration with VOCALOID
VOCALOID is a singing synthesis technology that enables
a creator to synthesize the vocal track of a song by inputting
the melody line with lyrics. In particular, Hatsune Miku [2]is the name of the most popular software package basedon VOCALOID and has a cute synthesized voice with anillustration of a cartoon girl. Most creators regard Hatsune
Miku as not only a musical instrument but also a pop star forwhom they create songs with lyrics to match her character.
Although only a few illustrations of Hatsune Miku wereoﬃcially provided, enormous numbers of illustrations havealready been drawn and shared by anonymous creators. Creators also started creating video clips of original songs featuring Hatsune Miku. Furthermore, some creators built
three-dimensional (3D) character models of Hatsune Miku
and created video clips containing 3D character animation.
The chain of such creation has been oﬃcially granted and encouraged by a unique license framework called Piapro Character License (PCL) that allows creators to use the Hatsune
Miku character for their own creations.
Creation of derivative works has also been supported bya reusable content-sharing web service called Piapro. Videoclips of derivative works on Nico Nico Douga were often
created in a manner similar to the remix culture in music[5]. Already published video clips are reused as materials tocreate new video clips. Creators usually welcome the reuseof their works because they are pleased to contribute to otherworks. The open collaboration of creating derivative works
has thus grown at a rapid rate.
After the success of Hatsune Miku, more than 30 software
packages based on VOCALOID, each having a diﬀerent voice
and character, have been released. Video clips of not only
Hatsune Miku but also other VOCALOID characters arealso uploaded in large numbers on Nico Nico Douga andreused in the open collaboration.

2.3

Key Factors

By analyzing the massive open collaboration that emerges
in the VOCALOID community on Nico Nico Douga, wefound that the following factors are essential.
• Availability of reuse: VOCALOID characters are oﬃcially allowed to be reused for creation. This directly
helps creators to create derivative works. It can also
contribute to an increase of active creators since reuseof content is often easier than creation of content fromscratch.
• Variety of expression: When music is reused, it canbe combined with various types of performances or
expressions, such as singing, dancing, animation, andmovies. This wide variety of derivative works makes
it easier for users to join the open collaboration, using their own skills or talents. Furthermore, the morediverse the content of video clips is, the wider the audience video clips can attract.
• Community-based encouragement: Time-synchronous
text comments on the Nico Nico Douga service fosters
communication among listeners and creators. Suchcommunication often encourages creators. Since Nico
Nico Douga has already hosted many derivative works,
there are many users who have already preferredderivative works from the beginning and can join themassive open collaboration.

2.4

Open Issues

Open collaboration within the VOCALOID community on
Nico Nico Douga is on the rise both in terms of quantity andvariety. Figure 3 shows the number of monthly publishedoriginal songs and their derivative works from September
2007 to August 2012. According to this figure, the numberof uploaded original songs and derivative works has been
increasing in general.
This wide variety of a huge amount of video clips makes
it more diﬃcult for users to grasp the whole picture. For
example, when a user listens to an original song, it is often
diﬃcult for the user to notice the existence of its derivative
works and play them back. If the user could easily accessderivative works, the user can have a better understandingof the massive open collaboration because derivative works
are good examples of the collaboration. The assistance ofbrowsing derivative works is thus important for the user tonot only find interesting video clips but also understand andrespect the collaboration in a deeper way. We believe thatsuch enhanced understanding of the open collaboration encourages both creators and consumers to participate in the
open collaboration.

Monthly number ofvideo clips
12000

of derivative works where a human singer covers a
VOCALOID song whose original main vocal was asynthesized singing voice. Some creators of an original song share the karaoke version without the main
vocal to encourage such derivative works.

derivative video clips(over 10,000 per month)

10000
8000
6000

(b) Dancing: A human dancer dances along with a
VOCALOID song. After a dancer choreographs dancemoves for a VOCALOID song, other dancers can reuse
them.

original songs

4000(over 2,000 per month)2000
0

Dec-2007

Dec-2008

Dec-2009

Dec-2010(c) Arranging and Performing: Human musicians arrange
and perform a VOCALOID song on musical instruments such as guitar, piano, and violin. Or, they covera VOCALOID song by using other VOCALOID voices.

Dec-2011

Figure 3: The monthly number of published original
songs and their derivative works during September
2007 – August 2012.

(d) Featuring 3D characters in music video: By using
a tool for creating three-dimensional (3D) computer
graphic animation, such as MikuMikuDance (MMD)1 ,some creators first design 3D character models, andother creators then reuse those models to create music
video clips. There are more than 10 diﬀerent models
for the Hatsune Miku character alone.

3. SONGRIUM
Songrium is a music browsing assistance service that facilitates the understanding of the massive open collaboration
within the VOCALOID community on the Nico Nico Douga
service. It automatically gathers information about original
songs and their derivative works that grow day after day. Itthen estimates relations between original songs and derivative works and visualizes them. Furthermore, Songrium has
a social annotation framework using the Arrow Tag to help
users encounter unfamiliar but interesting songs.
Songrium supports four functions: browsing original songs(Music Star Map), visualizing derivative works (Planet
View), annotating songs (Arrow Tag), and filtering content(Filter). The browsing and filtering functions facilitate discovery and understanding of songs. The annotation (ArrowTag) function allows users to reveal relations between original songs. The visualized relations and derivative works
provide a better user experience of retrieving and browsing
songs.

3.1 Original Songs and Derivative WorksSongrium continues to collect relevant information fromthe Nico Nico Douga service since we launched an alphaversion of Songrium in August, 2012. More than 97,000
VOCALOID original songs and 520,000 derivative works
have automatically been registered in Songrium.
In previous work [11], we analyzed and defined four categories of creative activity related to VOCALOID: Creating an original song; Covering an existing song by using
VOCALOID to synthesize the main vocal; Drawing picturesor creating graphics for a music video clip, and Editing different VOCALOID video clips for compilation albums andrankings.
In this paper, we propose the following six categories
of creative activity, which have emerged since the previous work. Now there are many derivative works in whichhumans are singing, dancing, and performing. In addition, VOCALOID songs have often been covered using
other VOCALOID voices diﬀerent from the original and
VOCALOID characters have appeared in non-music video
clips. This suggests that the personality of VOCALOID
characters has become established.
(a) Singing: This is currently the most popular category(e) Creating music video: Music video clips are often created by reusing other existing video clips or some materials such as 3D character models and singing voices.
(f) Others: Other derivative works include the karaokeversion of an original song, a medley of some original songs, and mashup video clips in which not only
original songs but also derivative works are reused.

3.2

Music Star Map Function

Music Star Map is a function that visualizes original songs
in a two-dimensional space where songs are automatically
mapped based on audio feature similarity. The position ofa song on the map is determined so that it can be locatedclose to songs having similar moods (i.e., audio features).
Figure 4 shows a screenshot of this function.
Well-known songs having many page views are displayedwith larger icons. Other songs are displayed with smaller
icons according to their number of page views. When a user
zooms into any area by double clicking, small icons becomelarger icons with thumbnail images from Nico Nico Douga.
The user can see other areas by dragging the mouse.

3.3

Planet View Function

When a user clicks one of the original songs on Music Star
Map, the selected song moves to the center. Its derivative
works then appear as colorful icons and rotate around theselected original song. We called this view Planet View.
Figure 5 shows a screenshot of the Planet View.
The oﬃcial embedded video player of the Nico Nico Douga
service is shown at the upper-right corner to play backa video clip of the selected original song. Our musiclistening interface with a chorus-search function for trial
listening, SmartMusicKIOSK [8], is shown below the embedded player. This SmartMusicKIOSK interface enables auser to jump to the chorus section and other key parts ofa song. It also visualizes the song structure consisting ofchorus and repeated sections. These functions make it easier for a listener to find the desired parts of a song, thereby
facilitating an enhanced listening experience.
1

http://www.geocities.jp/higuchuu4/index e.htm

Input form for keyword search

CAMusic Star Map
The map is zoomed inby double clicking on itDB
A list of video clips that were clicked recently
Slider for selecting the magniﬁcation

Figure 4: Screenshot of the “Music Star Map” interface for browsing original songs. (A) All original songs
are arranged in a two-dimensional space so that similar songs can be located close together. Any area can bezoomed in by double clicking and then scrolled to by dragging. Light blue lines indicate the personal recordof video clips that a user clicks on Songrium recently. (B) This personal record is also shown as thumbnail
icons at the bottom. (C) Songs can be searched by keywords. (D) The magnification ratio of the map canbe changed by a slider.

A list of Arrow Tags that represent relations of this songwith other songs is shown at the lower-right corner. The
user clicks one of the Arrow Tags in the list to jump to adiﬀerent original song. Or, the user can directly click one of
the Arrow Tags that are connected to the original song in
the center. Songrium then smoothly animates the transition
from the current song to the connected song by following theline of the clicked Arrow Tag. In contrast to the Music Star
Map that is useful for finding songs with similar moods,
Arrow Tags allow the user to find and encounter other typesof related songs that might have diﬀerent moods.
Once an original song has been selected, its derivative
works appear as circle icons around it. Each circle icon denotes a derivative work whose attributes are represented by
the orbit, size, color, and velocity of the icon. The radiusof the rotation represents the length of time between the
publishing date of the original song and its derivative work.
The size of each icon represents its popularity (page views),
and its velocity represents the degree of attention (a proportion of “favorites” to page views). The color indicatesthe category of each derivative work. When most icons havethe same color (i.e., most derivative works are in the same
category), the user can easily notice that this category dominates. In Figure 5, for example, there are many blue icons(Singing). In Figure 6 on the other hand, there are many
red icons (Dancing). The Planet View thus shows the powerof the original song to generate derivative works.
A user can also see a complete listed view of derivative
works. The user can play back one of them or sort them by
page views, categories, and publishing date.

3.4

Arrow Tag Function

Arrow Tag is a directional named link from one song to another song. It describes a relation between original songs innatural language. Arrow Tag A from song1 to song2 means“Song1 has an ‘A’ relation to Song2”. Songrium enables auser to annotate various relations among original songs using
Arrow Tags. It can encompass non-directional relations between songs (e.g., same genre, same theme) in the same way
as traditional tagging methods, but it also has the specificadvantage of making a Songrium user aware of directional
relations between two songs (e.g., a new arrangement or a
response to another song).
Some relations are directional (e.g., same genre, same
theme), and others are non-directional (e.g., answer song,arrange). Since some of non-directional relations can be described by existing representation methods, the Arrow tag
includes information of its direction.
The data structure of each Arrow Tag is a triple of theform (“Song as the starting point”, “Name of relation”, “Songas the end point”). Songs annotated by Arrow Tags construct a directed graph where each node is a song and eachedge is an Arrow Tag. More than two Arrow Tags can be
put between songs. The same Arrow Tag can also be put
between songs bidirectionally, for example.
In the following, we present examples of Arrow Tags thathave been added by Songrium users. We can find that various Arrow Tags are generated. The Arrow Tag does not
restrict the type of represented relations. In general, social
tagging can represent both a subjective and objective phenomenon [26]. Our Arrow Tag can also be used for implicit,

Distance: Publishing date

E

Video Player

F

Playback interface
for trial listening(SmartMusicKIOSK)G

A list of Arrow Tags

Size: Page views
Color: Derivative category
Speed: Proportion of “favorites” to page views
Figure 5: Screenshot of the “Planet View” interface. After selecting an original song on Music Star Map, a user
can see its derivative works rotating around the selected song in the center. (E) The embedded video playerof Nico Nico Douga for video playback. (F) Playback interface for trial listening (called SmartMusicKIOSK).
(G) Arrow tags from and to this song are shown. (H) An explanation of the icon for each derivative work.
The distance from the center represents the publishing date. The latest work is in the outermost orbital.
The size of each icon reflects the number of its page views. The color indicates one of the following categories(see Section 3.1): Blue (Singing), Red (Dancing), Green (Arranging and Performing), Purple (Featuring 3D
characters in music video), Yellow (Creating music video), and White (Others).

explicit, subjective, and objective relations. In fact, it is important that they can be mixed. Users can thus freely tag
and share an Arrow Tag. If somebody invents a new usefullabel for Arrow Tag, other users will start reusing it.
• “Creator’s next song”: This represents a relation fromthe current song to the next released song by the same
creator.
• “Want to listen after this”: When a user finds a songthat is suitable to listen to after the currently playing song, the user can put this Arrow Tag as a music
recommendation.
• “Similar themes in the lyrics”: When the lyrics of twosongs have similar themes, this Arrow Tag can be put.
This is useful for listening to only songs having similar
lyrics.
• “Respect this song”: If a creator of the current songrespects another song, this Arrow Tag can be put fromthe current song as the starting point to the respectedsong as the end point.
Figure 6 shows how to input a new Arrow Tag. Whena user adds an Arrow Tag between two original songs,
Songrium shows excerpts of popular existing Arrow Tags
as suggestions or a complete list of all Arrow Tags.
In our current implementation, Songrium only allows
users to put Arrow Tags only between original songs. A
derivative relation between an original song and a derivative
work is automatically estimated and registered in Songrium,

so far. However, the Arrow Tag has the potential to represent various relations not only among original songs but also
among derivative works. We will discuss the future use of
Arrow Tags in Section 6.2.

3.5

Filter Function

Filter is a function that enables a user to make a personal short list of video clips selected by criteria or rules
using categories and indicators such as genre, popularity,and publishing date. Traditional tag search or bibliographic
sort has already been useful in general, but the user has
to input queries every time. Complex rules are not usuallyused in a query because it is too diﬃcult for some users todesign them.
This Filter function can easily produce a short list of video
clips based on page views, the number of users who put “favorite”, tags, publishing dates, and so on. The user can simply choose a preset or saved “filter” to narrow down the listof video clips. One filter could consist of multiple rules suchas “page views higher than 1,000” and “publishing date wasless than 1 week ago”. Since general useful filters have already been pre-defined, the user can simply customize themor generate a new filter by designing rules. Furthermore, if
the user likes the results, the user can name and save thepersonalized filter for future reuse.

4.

IMPLEMENTATIONSongrium consists of three components: a web crawler,
a web server, and a user interface. The web crawler automatically checks updated music video clips related toSelection of a target song for Arrow Tag
I

Table 1: Classification results of 522,875 derivative
works. Since some derivative works have multiple
categories, the sum of classifications is greater than
the total number of derivative works.
category(a) Singing(b) Dancing(c) Arranging and Performing(d) Featuring 3D characters(e) Creating Music video(f) OthersJ# of works
384,726
22,76422,99518,991
7,826
70,874

Input form of Arrow Tag
Figure 6: Screenshot of the input form to generatea new Arrow Tag. (I) A user searches video clips by
keywords and selects one of them as the end point of
a new Arrow Tag. (J) The user then types in a label
and chooses the direction of an Arrow Tag. Existing
Arrow Tags are suggested as input aids below the
input form.

VOCALOID on Nico Nico Douga every day. It classifies
them using the technique described in Section 4.1, and estimates their chorus and repeated sections using the method
described in Section 4.2. The web server generates the MusicStar Map as described in Section 4.3, maintains a databasefor video clip metadata and Arrow Tags, and works as a
website that provides the Songrium user interface. The userinterface is implemented using HTML5, SVG, JavaScript,
the JavaScript library D3.js, and the embedded video playerof the Nico Nico Douga service. Songrium can be used on thelatest web browsers including Chrome, Safari, and Internet
Explorer.

4.1 Classification of Video Clips
Every music video clip on Songrium is automatically classified into original song or derivative work. Because Nico
Nico Douga supports social tags for each clip and the “Original Song” tag is usually put on original songs on Nico Nico
Douga, we can rely on this tag. However, even if some original songs do not have this tag, Songrium correctly classifies
them by crawling a set of related web sites to automaticallygenerate the “white list” of VOCALOID original songs. In
order to check accuracy of our method, we randomly selected
100 videos that were classified as VOCALOID original songs
and reviewed them manually. Of these, 89 videos were correct, 6 videos were alternative versions of original songs, and5 videos were derivative works or not songs.
In the case of derivative works, these can be readily identified when the description text of the video clip includes a
hyperlink to the original video clip from which it was derived. These hyperlinks almost always exist on Nico Nico
Douga since users like to acknowledge the original video clip.
When a new original VOCALOID song is added to Nico
Nico and incorporated into Songrium, its relations to otheroriginal songs are analyzed and some Arrow Tags can be
automatically assigned. In the current implementation,
Songrium can only put the Arrow Tag “Creator’s next song”
to link two songs from the same creator. This connection

is made using the “white list” and the video clips with acreator’s unique ID number.
Likewise, when a derivative work is incorporated, its relation to the original song is automatically estimated. The
derivative works are classified into one of the six predefined categories described in Section 3.1. With the exceptionof category (f) Others all the remaining categories are extremely popular and have their own unique social tags on
Nico Nico Douga. Using these tags Songrium can make areliable classification of derivative works. Table 1 shows theresult of the classification of 522,875 derivative works.
Moreover, Songrium enables users to easily report an error in any of the above classifications to improve the user
experience.

4.2 Estimation of Chorus Sections
Chorus sections and repeated sections are estimated byusing the audio-based chorus-section detection method RefraiD [8], which was also used as the back-end for the original
SmartMusicKIOSK interface. By analyzing relationships
between various repeated sections, the RefraiD method can
detect all the chorus sections in a song and estimate boththe start and ends points of each section. It can also detect
modulated chorus sections, which are especially common in
pop music.

4.3

Generation of Music Star Map

The position of each original song is mapped to a twodimensional space of the Music Star Map based on analysis
of audio features. Although any feature vector designed forcomputing music similarity could be used, in our current implementation, we use a 35-dimensional feature vector, which
is obtained using the MARSYAS software framework [28]that automatically analyzes the mood of each song. The effectiveness of this feature vector has already been confirmedin the context of music genre classification. The feature vector consists of the mean and variance of local spectral features (centroid, rolloﬀ, flux, and zero-crossings) calculated
across the entire song (8 dimensions); the mean and variance of average values of mel-frequency cepstral coeﬃcients(MFCC) across the entire song (26 dimensions); and thetempo in the chorus section (1 dimension).
To determine the (x-y) position of a song on the MusicStar Map, its 35-dimensional feature vector is projected onto
two dimensions using principal component analysis (PCA),
where only the first two components are retained.

4.4

Optional Web Browser Extension

Although Songrium is a web service that can immediately
be used on a web browser without installing any additional
components, we have also developed an optional browser
extension designed for heavy users of both Songrium andNico Nico Douga. After installing the extension, users haveaccess to some Songirum functions directly on Nico Nico
Douga. For example, when the user watches the video clipof an original VOCALOID song on Nico Nico Douga, ourextension inserts the SmartMusicKiosk interface below thevideo clip so that the user can jump to the chorus section.
The user can also see the Planet View and use Arrow Tags on
the Nico Nico Douga service, thereby facilitating a seamlessexperience of both services.

5. EXPERIENCES WITH SONGRIUM
The alpha version of the Songrium service was released
to the public at http://songrium.jp on August 7th, 2012.
In addition to the web service, the Songrium extension for
Google’s Chrome browser was then released on February
28th, 2013. As of June 2013, over 97,000 original songs and
520,000 derivative works have been registered in Songrium.
Over 7,000 users have visited our web site and over 1,700
users have installed the browser extension. Many users havemade favorable comments about Songrium on social media, and have indicated their excitement in obtaining a newmeans to find music content with open collaboration. One
user wrote that “Songrium aﬀects my style of watching andlistening to VOCALOID video clips”.

6. DISCUSSION
While research dealing with a public web service for musicbrowsing assistance based on visualization of massive open
collaboration has not been pursued in the past, there havebeen various approaches related to open collaboration andmusic browsing. In the following, we introduce related workand then discuss how Arrow Tags of Songrium could open
up new possibilities of music content in the future.

6.1 Related Work6.1.1 Encouraging Open Collaboration
Antin et al. [1] highlight the importance of conducting
research into open collaboration across diverse platforms anddomains and building knowledge through analysis. In fact,
not only well-known platforms such as Wikipedia and open
source software (OSS), but also other platforms and domains
were analyzed from the viewpoint of open collaboration [4,5, 18, 22]. Our research also contributes to this diversity byanalyzing a new style of music content creation.
Most previous research into such open collaboration, however, has mainly focused on the analysis of collaboration, noton the direct assistance of users on the basis of the analyzed
results. For example, the collaboration in social networks
was analysed in [13, 14, 5], and the collaboration outsideof social networks was analyzed in [19, 5, 6]. Even if thoseprevious approaches visualized some relations within open
collaboration, they did not help users collaborate or enjoy
the results of collaboration on the basis of the analysis andvisualization.
A key aspect of our research is not just to analyze and visualize open collaboration but to provide a means for usersof Songrium to engage in a deeper and more meaningful

collaboration with music content and other users. In thisway, we believe this research to be the first of its kind. Furthermore, the open collaboration we analyze and visualize
addresses reuse of content into multiple forms as describedin section 3.1. This is unique since reuse of content in a single form was analyzed in most previous approaches dealing
with reuse network of content creation [5, 6].

6.1.2

Assisted Music Browsing

Most previous research into interactive music browsing
has eﬀectively used visualizations to explore musical collections [3, 20, 29, 21, 9, 25]. For example, as we showed in theMusic Star Map (see Section 4.3), visualization of a musiccollection in a two-dimensional plane is often attempted [3,20, 29, 25]. Interactive interfaces are also important for user
experiences; the Musicream interface [9] assists a user in discovering songs and the MusicRainbow interface [21] assists
a user in finding artists. Songrium oﬀers users additional
functionality, where they can interact with, and contributeto, the Music Start Map through the use of Arrow Tags.
This collaborative aspect of Songrium as well as the visualization of derivative works is essential for music browsingof massive open collaboration, and is beyond the scope ofprevious approaches.
The Arrow Tags we proposed are also related to previous studies that dealt with music relations in the context of
the Semantic Web [23, 17]. Since the Semantic Web focuseson the construction of structured data, the consistency ofsemantics of relations is important [12]. However, within
Songrium we choose to allow the Arrow Tags to be inconsistent because we believe it is more important that non-expert
users feel free to add any Arrow Tag to any music relationof interest to them. This leads to the interesting prospect ofa wide variety of semantics of relations. The purpose of theArrow Tags is thus original.

6.2

Future Use of Arrow Tags

This section discusses a vision of the potential usage of theArrow Tag in the future by showing how Arrow Tags could
be used to augment the user experience within a massive
open collaboration. If users find Arrow Tags connected to
one of their favorite songs, they can, of course, follow the Arrow Tags to listen to and discover other original songs. This
gives users the opportunity to encounter unfamiliar but potentially interesting songs and can cultivate a deeper understanding of songs and their musical relations. In addition,
we describe five use cases, shown in Figure 7, which could
be achieved through widespread usage of Arrow Tags.
1. Before taking the time to experience (e.g., listen toor watch) a particular piece of music content, a user
could understand its relations to other music content
from the Arrow Tags and see whether the user would
be interested in it. In Figure 7-(1), for example, themusic content 1 is tagged with Arrow Tag ‘A’ fromthree other pieces of music content. If a user has already been interested in Arrow Tag ‘A’, the presenceof Arrow Tag ‘A’ pointing to music content 1 could be
suﬃcient by itself to attract the interest of the user
towards this content 1.”
2. Once a user becomes familiar with a certain relation
thanks to the use of Arrow Tags, the user could easily
notice this relation between pieces of music content.

A
A

1

a huge variety of Arrow Tags could emerge and go
through a form of “natural selection”. Arrow Tags
could thus evolve towards creating a better representation of relations between music content as in Figure
7-(4).

A(1) A user can guess the value of music content
from its relations before experiencing its content.

2??

5. The widespread usage of a certain Arrow Tag could
inspire the creation of a new piece of music content.
In Figure 7-(5), a creator who becomes familiar withthe usage of Arrow Tag ‘A’ could create music content
2 so that it could potentially have the relation ‘A’ withmusic content 1 and the user could put the Arrow Tag‘A’ later. The Arrow Tags could thus contribute tocontent creation.

2

A
1

1

(2) A user can notice an unconscious relation
between music content.

2
A

A

1

4

2

4

6

A

B

B

B

3

1

3

5

7.

(3) A user can get interested in a group ofmusic content having the same relation.

A
B

1

C

2
3

1

2

A
D

E

3
4

(4) A better representation of relations
between music content can emerge.

A
?

1

A

Beyond these five use cases we fully expect and hope to
see other uses of Arrow Tags, which naturally emerge fromthe open collaboration in Songrium.

12(5) A relation can inspire the creation ofa new piece of music content.
Figure 7: Five use cases that could be brought
through widespread usage of Arrow Tags.

In Figure 7-(2), for example, a user could notice an
unlabeled hidden relation (i.e., A ) between music
content 1 and 2 and put Arrow Tag ‘A’ to mark thisrelation.
3. A user could be interested in a group of music content
having the same relation and experience this group together. At the left of Figure 7-(3), music content 1–4
are associated by the Arrow Tag ‘A’, which could help
the user become aware of this group of music content.
The Arrow Tag could thus be used as a method forgrouping content. At the right of Figure 7-(3), on theother hand, a user could notice that three pairs of music content have the same Arrow Tag ‘B’ and become
interested in exploring these pairs.
4. If various users are encouraged to freely invent andshare new Arrow Tags and delete inappropriate ones,

CONCLUSIONS

We have described Songrium, a music browsing assistanceservice in which the Music Star Map and Arrow Tag functions enable the interactive visualization and annotation ofrelations among original songs. The Planet View function
also enables the interactive browsing of relations among
derivative works and their original song, which are results of“massive” open collaboration. Moreover, the Filter function
facilitates the creation of a personalized list of music content. Songrium makes a social contribution by providing the
public web service that helps both creators and consumers
understand the huge reuse network within the VOCALOID
community. We hope that such enhanced understanding of
the open collaboration further facilitates more massive open
collaboration in the future.
The academic contribution of this study is to show how
the results of analyzing the open collaboration can be used
for the assistance of the further open collaboration. While
the majority of existing work on open collaborations merely
analyze the open collaboration, we believe that the topic
of assistance within the open collaboration is of particularimportance and should also be addressed much more in the
future. Interactive visualization is one of the most eﬀective
ways to assist users, and we propose Songrium as a usefulexample.
Open collaboration creates a lot of user-generated dataand organizing this data is an important issue. Within
Songrium our proposed solution is the Arrow Tag which
is not used to name and mark music content directly, butrather to mark a relation between diﬀerent pieces of music
content. We consider the Arrow Tag to be a simple, yet powerful way to organize relations of any type of content. Since
Arrow Tags can be created and deleted by users and content
creators, they can evolve to provide a rich environment forunderstanding music relations.
We will continue to run the Songrium service and improveit based on user feedback. Although Songrium has alreadysupported typical categories of derivative works, we plan toanalyze other emerging categories of derivative works arising from open collaboration as it is a significant and uniquecharacteristic of the VOCALOID community on Nico Nico
Douga. Furthermore, we hope that the widespread use ofuser-generated Arrow Tags will give us new insight for understanding the whole picture of the huge reuse network in
the VOCALOID community.

8. ACKNOWLEDGMENTS
We thank Keisuke Ishida for the web service implementation of Songrium. We also thank anonymous users ofSongrium for editing social annotations. This work was supported in part by CREST, JST.

9. REFERENCES[1] J. Antin, E. H. Chi, J. Howison, S. Paul, A. Shaw, andJ. Yew. Apples to oranges?: comparing across studies
of open collaboration/peer production. In Proc. of
WikiSym ’11, pages 227–228, 2011.
[2] Cabinet Oﬃce, Government of Japan. Virtual idol. In
Highlighting JAPAN, volume 2, pages 24–25. 2009.
[3] P. Cano, M. Kaltenbrunner, F. Gouyou, and E. Batlle.
On the use of fastmap for audio retrieval andbrowsing. In Proc. of ISMIR ’02, pages 275–276, 2002.
[4] G. Cheliotis. From open source to open content:
Organization, licensing and decision processes in open
cultural production. Descision Support Systems,
47:229–244, 2009.
[5] G. Cheliotis and J. Yew. An analysis of the social
structure of remix culture. In Proc. of C&T ’09, pages
165–174, 2009.
[6] K. Eto, M. Hamasaki, K. Watanabe, Y. Kawasaki,
and T. Nishimura. Modulobe: a creation and sharing
platform for articulated models with complex motion.
In Proc. of ACE ’08, pages 305–308, 2008.
[7] A. Forte and C. Lampe. Defining, understanding, andsupporting open collaboration: Lessons from theliterature. American Behavioral Scientist, 1 2013.
[8] M. Goto. A chorus-section detection method formusical audio signals and its application to a music
listening station. IEEE Transaction on ASLP,
14(5):1783–1794, 2006.
[9] M. Goto and T. Goto. Musicream: Integrated
music-listening interface for active, flexible, andunexpected encounters with musical pieces. IPSJ
Journal, 50(12):2923–2936, 2009.
[10] M. Hamasaki, H. Takeda, T. Hope, and T. Nishimura.
Network analysis of an emergent massively
collaborative creation community: How can people
create videos collaboratively without collaboration? In
Proc. of ICWSM ’09, 2009.
[11] M. Hamasaki, H. Takeda, and T. Nishimura. Network
analysis of massively collaborative creation ofmultimedia contents - case study of hatsune miku
videos on nico nico douga -. In Proc. of uxTV ’08,
pages 165–168, 2008.
[12] T. Heath and C. Bizer. Linked Data: Evolving the Web
into a Global Data Space. Morgan & Claypool, 2011.
[13] B. Heller, E. Marschner, E. Rosenfeld, and J. Heer.
Visualizing collaboration and influence in the
open-source software community. In Proc. of MSR ’11,
pages 223–226, 2011.
[14] A. Jermakovics, A. Sillitti, and G. Succi. Mining andvisualizing developer networks from version control
systems. In Proc. of CHASE ’11, pages 24–31, 2011.
[15] K. Kaji and K. Nagao. Mixa: A musical annotation
system. In Proc. of ISWC ’04, 2004.
[16] H. Kenmochi and H. Ohshita. Vocaloid – commercial
singing synthesizer based on sample concatenation. In

Proc. of Interspeech ’07, pages 4011–4010, 2007.
[17] S. Kolozali, M. Barthet, G. Fazekas, and M. Sandler.
Knowledge representation issues in musical instrument
ontology design. In Proc. of ISMIR ’11, pages
465–470, 2011.
[18] C. Lampe, R. Wash, A. Velasquez, and E. Ozkaya.
Motivations to participate in online communities. In
Proc. of CHI ’10, pages 1927–1936, 2010.
[19] F. Mödritscher, W. Taferner, A. Soylu, and
P. De Causmaecker. Visualization of networked
collaboration in digital ecosystems through two-mode
network patterns. In Proc. of MEDES ’11, pages
158–162, 2011.
[20] E. Pampalk and S. Dixon. Exploring music collections
by browsing diﬀerent views. Computer Music Journal,
28(2):49–62, 2004.
[21] E. Pampalk and M. Goto. Musicsun: A new approachto artist recommendation. In Proc. of ISMIR ’07,
pages 101–104, 2007.
[22] K. Panciera, M. Masli, and L. Terveen. ”how should itowithout getting killed?”: motivation
go fromand benefits in open collaboration. In Proc. of
WikiSym ’11, pages 183–192, 2011.
[23] Y. Raimond, S. Abdallah, M. Sandler, and F. Giasson.
The music ontology. In Proc. of ISMIR ’07, pages
417–422, 2007.
[24] D. Riehle, J. Ellenberger, T. Menahem,
B. Mikhailovski, Y. Natchetoi, B. Naveh, andT. Odenwald. Open collaboration within corporations
using software forges. IEEE Software, 26(2):52–58,
2009.
[25] M. Schedl, C. Hoglinger, and P. Knees. Large-scalemusic exploration in hierarchically organizedlandscapes using prototypicality information. In Proc.
of ICMR ’11, pages 17–20, 2011.
[26] S. Sen, S. K. Lam, A. M. Rashid, D. Cosley,
D. Frankowski, J. Osterhouse, F. M. Harper, andJ. Riedl. tagging, communities, vocabulary, evolution.
In Proc. of CSCW ’06, pages 181–190, 2006.
[27] D. A. Shamma, J. Yew, L. Kennedy, and E. F.
Churchill. Viral actions: Towards predicting viral
videos using synchronous sharing behaviors. In Proc.
of ICWSM ’11, 2011.
[28] G. Tzanetakis and P. Cook. Marsyas: A framework foraudio analysis. Organized Sound, 4(3):169–175, 2000.
[29] R. van Gulik, F. Vignoli, and H. van de Wetering.
Mapping music in the palm of your hand, explore anddiscover your collection. In Proc. of ISMIR ’04, 2004.
[30] D. Yamamoto and K. Nagao. ivas: Web-based video
annotation system and its applications. In Proc. of
ISWC ’04, 2004.
[31] J. Yew, D. A. Shamma, and E. F. Churchill. Knowing
funny: genre perception and categorization in social
video sharing. In Proc. of CHI ’11, pages 297–306,
2011.