2015 IEEE International Symposium on Multimedia

Songle Widget: Making Animation and Physical Devices
Synchronized with Music Videos on the Web
Masataka Goto

Kazuyoshi Yoshii

Tomoyasu Nakano

National Institute of Advanced Industrial
Science and Technology (AIST), Japanm.goto@aist.go.jp

Kyoto University,Japan
yoshii@kuis.kyoto-u.ac.jp

National Institute of Advanced Industrial
Science and Technology (AIST), Japant.nakano@aist.go.jp

Abstract—This paper describes a web-based multimedia development framework, Songle Widget, that makes it possible tocontrol computer-graphic animation and physical devices suchas lighting devices and robots in synchronization with musicpublicly available on the web. To avoid the difculty of timeconsuming manual annotation, Songle Widget makes it easy
to develop web-based applications with rigid music synchronization by leveraging music-understanding technologies. Fourtypes of musical elements (music structure, hierarchical beatstructure, melody line, and chords) have been automaticallyannotated for more than 920,000 songs on music- or videosharing services and can readily be used by music-synchronizedapplications. Since errors are inevitable when elements areannotated automatically, Songle Widget takes advantage ofa user-friendly crowdsourcing interface that enables users tocorrect them. This is effective when applications require errorfree annotation. We made Songle Widget open to the public,
and its capabilities and usefulness have been demonstrated inseven music-synchronized applications.
Keywords-music synchronization; music understanding; webservice; multimedia control

I. I NTRODUCTION
Since music can be easily and effectively combined withvarious types of content, it is often used when showing
images, movies, and stories, and controlling physical devices
such as robots and lighting devices. Synchronization is
crucial when music is combined with other content, andto rigidly synchronize animation or physical devices withmusic, people usually have to annotate temporal positions oftarget events in the music. This manual annotation is timeconsuming, however.
Automatic music analysis is therefore useful for rigid
synchronization. While there are music visualizers built into
existing media players that can show music-synchronized(music-sync) animation of geometric patterns, their music
analysis is usually based on the amplitude or spectrogramof audio signals. Such analysis is too basic to reect various
musical elements such as musical beats, bar lines (downbeats), chorus sections, and chord changes.
The goal of our research is to provide a web-based open
framework that enables people to control virtual or physicalobjects in synchronization with the playback of songs ormusic video clips on music- or video-sharing services on
the web. Although it has become popular for people to enjoy
music by watching a music video clip on the web, the fullpotential of the ever-increasing number of songs on the web
978-1-5090-0379-2/15 $31.00 © 2015 IEEE
DOI 10.1109/ISM.2015.64

Figure 1. Overview of Songle Widget framework (http://widget.songle.jp)
using Songle web site (http://songle.jp).

has yet to be realized. We want to make it possible to musicsynchronize almost anything. While playing back a song on
the YouTube video-sharing service, for example, we wantpeople to enjoy various types of computer-graphic animationin which the motion, size, and color of various graphicalobjects on a web browser change in ways coordinated withthe music being played back. We also want to control
physical devices such as lighting devices and humanoid
robots through such web-driven music synchronization.
To enable easy development of these kinds of music-sync
web applications, we propose a new web-based framework
called Songle Widget (Figure 1). To harness the power ofmusic on the web, Songle Widget is implemented by using
our existing web service called Songle (http://songle.jp)[1] that leverages music-understanding technologies to automatically analyze songs publicly available on the web
and annotate four important types of musical elements(music structure, hierarchical beat structure, melody line,
and chords) as shown in Figure 1. Songle Widget can use
those annotation results to trigger changes in the motion and85

color of animation, or in the control of physical devices.
Given a wide variety of music, a drawback of musicunderstanding technologies is that errors are inevitable.
Some users and music-sync applications require error-freeannotations for high-quality music synchronization in public
performances and presentations. To overcome this drawback,Songle Widget takes advantage of a Songle’s crowdsourcing
interface that enables users to correct errors in the annotation results. Songle Widget is thus considered a dynamic
evolving framework that provides immediate access to upto-date musical elements improved by users. This featureis original to our framework and has not been achievedby state-of-the-art music-understanding technologies [2]–[4],music-analysis web APIs such as the Echo Nest API (http://developer.echonest.com) and the AcousticBrainz project(http://acousticbrainz.org), or other systems for generatingmusic-sync display, animation, or videos [5]–[7].
We have already made the Songle Widget framework open
to the public (http://widget.songle.jp). To demonstrate its
capabilities and potential, in this paper we present sevenmusic-sync applications. While playing back a music video
clip on YouTube, for example, those web applications enablehumanoid robot devices or animated human-like computergraphic characters to dance to music, they control multiple
lighting devices projecting various types of music-sync
lighting patterns onto a stage-like space, and they show
graphical objects whose changes in motion, size, and color
are synchronized with changes in music.

Figure 2. Screenshots of Songle Widget user interface with different appearances. This is equipped with the SmartMusicKIOSK interface proposedby Goto [8], which shows the music structure consisting of chorus sections(the top row) and repeated sections (the ve lower rows). The coloredsections in each row indicate similar (repeated) sections.

Since using the embedded YouTube/Niconico player todevelop music-sync applications is not easy and requires advanced engineering skills and careful efforts, Songle Widget
provides the event-driven architecture in which programmers
can just write a JavaScript code for each event, such as eachmusical beat in a bar, the bar line (downbeat), and the chord
change. The beginning or end of each chorus or repeatedsection can also be used as an event. In controlling physicaldevices such as robots (or even computer graphics), however,latency is problematic because the actual robot motion isusually delayed for tens or hundreds of milliseconds after
being triggered by a motor-control signal. Songle Widget
therefore supports a timing-offset mechanism in which a
code for each event can be triggered before the actual event.
For high-quality synchronization, we can take full advantage of the crowdsourcing error-correction interface on
Songle. With this interface, users can make corrections bycandidate selection or direct editing. Any error corrections
made to the musical elements can be instantly reected on
all applications using Songle Widget.

II. I SSUES IN W EB - BASED M USIC S YNCHRONIZATIONMusic synchronization on the web is difcult. When auser plays back a music video clip on popular video-sharing
services, such as YouTube (http://www.youtube.com) andNiconico (http://www.nicovideo.jp/video top), or uses theirofcial embedded player, a user program on a web browser
cannot receive and analyze audio signals of the clip. Realtime on-the-y audio analysis for music synchronizationis therefore not feasible and there is no web framework
dedicated to it.
In this paper we propose a new approach for music synchronization, one that executes music-understanding technologies on music on the web in advance to obtain four
important types of musical elements (Figure 1):(i) Music structure (chorus sections and repeated sections)(ii) Hierarchical beat structure (musical beats and bar lines)(iii) Melody line (fundamental frequency (F0) of the vocalmelody)(iv) Chords (root note and chord type)
We represent each musical element as a series of timestamped events (e.g., beat positions) so that Songle Widget
can compare the time stamps with the current playback
position to trigger a user program. Songle Widget uses theofcial embedded YouTube/Niconico player and its API
to get the current playback position (elapsed time) while
playing back a video clip. Music-sync applications canthus synchronize animation and physical devices with music
videos on the web.

III. S ONGLE W IDGET: W EB - BASED D EVELOPMENTF RAMEWORK FOR M USIC - SYNC A PPLICATIONS
Figure 2 shows the user interface of Songle Widget, whichallows a compact dedicated player to be embedded in any
web page for music-sync web applications. The outstanding
features here are that it enables music-sync applications to
instantly access the musical elements for more than 920,000
songs, which was hitherto difcult to achieve without musicunderstanding technologies.
To facilitate the development based on Songle Widget, we
provide a template to write the JavaScript source code using
the Songle Widget API so that programmers can simply add
and modify codes for each event. For example, if a user
code for showing a visual effect is written for the event
corresponding to the bar line, its effect is automaticallyshown at the beginning of each bar. Programmers could
also write an additional code to change all visual effects
into more intense ones during chorus sections. Programmers
without knowledge of music-related programming can thusachieve music-sync control quite easily.
The implementation overview is shown in Figure 3.
Songle Widget was carefully designed and implemented sothat it can be embedded into any web-based music-sync

86Figure 4. Photographs showing stage lighting linked to music by using
Songle Widget.

Figure 3. Overview of Songle Widget implementation (in the case of using
beat, chorus, and chord events with the YouTube embedded video player).

Figure 5. Photographs of music-sync robot dancing controlled by V-Sido
x Songle.

application without harmful side effects. We therefore chosean IFrame-based sandbox implementation that can execute
our JavaScript code in a controlled environment isolated
from the user application. For the sandbox implementationin JavaScript, we had to implement all necessary functions
such as input sanitizer and DOM (Document Object Model)
manipulation by ourselves without using external libraries
such as jQuery and underscore.js.
As shown in Figure 3, Songle Widget consists of two
components: player and API. The Songle Widget player
generates the user interface shown in Figure 2, manages userinteractions on a web browser, and provides an encapsulation
wrapper of different embedded video players and music
players. On the other hand, the Songle Widget API serves as
the programming interface for user applications and handles
all events to trigger user programs. Those user programs
should be registered and bound to events (musical elements)
in advance by using this API. To achieve interdomain
communication over the sandbox and IFrame, we use theHTML5 Web Messaging mechanism.

B. Three-dimensional Animated Dancing Characters
In December 2012 Crypton Future Media, Inc. used
Songle Widget to let visitors of their site watch threedimensional computer-graphic characters dancing to music
through WebGL rendering.
This music-sync application also has a touchpad-like
display with buttons labeled with the names of chords.
These buttons light up in synchronization with the chord
information from Songle Widget, and users can push any ofthese buttons to hear synthesized voices singing that chord.
C. Music-sync Lighting
In 2013 we started using Songle Widget to link realworld physical devices — lighting devices — to music.
This music-sync application supports the control of various
lighting devices compatible with the DMX512 standard. It
enables physical lighting devices to be linked to the musicalelements of any song being played on Songle Widget andto be controlled accordingly. As shown in Figure 4, varioustypes of lighting linked to music were projected in a stagelike setting.

IV. C ASE S TUDY: M USIC - SYNC A PPLICATIONS
Since we have already made the Songle Widget framework open to the public, various music-sync applications
using Songle Widget have been developed by us and by
third parties. Seven of them are chosen to represent example
applications.

D. Melvie: VJ Service for Coloring Music with Videos
A researcher Makoto Nakajima used Songle Widget incollaboration with us to develop the video jockey (VJ)
web service Melvie (http://melvie.jp) that was opened tothe public in June 2014. Melvie renders different sourcesof short video clips without audio after mixing them andapplying various special effects such as overlaying, zooming,tiling, and color change in synchronization with the music
playback.

A. Two-dimensional Music-sync Animation
The Songle Widget framework has been used since August 2012 to draw music-sync animation in the backgroundof various web pages including personal homepages andblogs. We provide sample source codes for music-sync background animation in which each bar line (downbeat) generates a new expanding and disappearing pattern of geometricshapes (circles, triangles, or squares), the beginning of achorus or repeated section generates several simultaneous
expanding and disappearing patterns with different colors,and each chord change changes the background color of theembedding web page.
In August 2012 Songle Widget was used by Crypton
Future Media, Inc. to let visitors of their site watch a twodimensional animated character dancing to music.

E. V-Sido x Songle: Real-time Control of Music-sync RobotDancing
A roboticist Wataru Yoshizaki used Songle Widget incollaboration with us to develop a music-sync robot control
system, called V-Sido x Songle, that automatically switches
several different predened dance motions according to themusic structure and hierarchical beat structure of any song
registered to Songle. By using a joystick or tablet, people canchange motions on the y while the robot is dancing. Such87

Table I
E VENTS ( MUSICAL ELEMENTS ) USED IN MUSIC - SYNC APPLICATIONS .
Application
Two-dimensional animation
Three-dimensional characters
Lighting devices
MelvieV-Sido x Songle
Songrium3D
Photo x Songle

Figure 6. Screenshots of Songrium3D, the music visualizer for generating
three-dimensional music-sync animation in real time.

exible music-sync robot control had not been achieved
before. In January 2015 Asratec Corp. and AIST publisheda press release and showed that this system can makethree different types of robots dance in unison as shownin Figure 5.

BeatV
V
V
V
V
V
V

Structure (Chorus)
VV
VV
V
V
V

ChordVVV

VI. C ONCLUSIONS
We have described the Songle Widget framework thatmakes it easy for people to develop and enjoy musicsync applications such as music-sync animation, music-sync
dancing of robots or animated characters, and music-sync
lighting. Since more than 920,000 publicly available songs
have already been analyzed by using music-understanding
technologies, users can immediately choose among themfor music-sync applications. To use error-free annotationof musical elements, users can also make voluntary error
corrections on the web-based interactive editor.
The concept of the proposed framework is general anduseful not only for music and its analysis but also for othertypes of multimedia content, such as images and videos,
and their analysis. Future work will therefore include thedevelopment of a web-based development framework forapplications driven by a mixture of any multimedia content
on the web.

F. Songrium3D: Music Visualizer Featuring Threedimensional Music-sync Animation
Songle Widget has been used in an advanced musicvisualizer, called Songrium3D (http://songrium.jp/map/3d/)[9], that features three-dimensional music-sync animation inwhich visual effects and the motions of various objects aretriggered by events in the music structure and hierarchicalbeat structure of a music video clip as shown in Figure 6.
G. Photo x Songle: Music-sync Photo Slideshow
In December 2014 Songle Widget was used to develop
a web service Photo x Songle (http://photo.songle.jp) thatenables users to generate photo slideshows. A keyword orphrase entered by a user is used to retrieve relevant photos
on the web by using Google Images. Those photos are thenshown as animated slideshows during the playback of any
song registered to Songle. A new photo usually appears at
the beginning of each bar, but during chorus sections thenew photos appear at every beat and with more vivid motion
effects.

ACKNOWLEDGMENTS
The authors would like to extend their appreciation toTakahiro Inoue for the player/API/application implementation of the Songle Widget framework, and Utah Kawasaki
for the server/client implementation of the Songle webservice. This work was supported in part by CREST, JST.
R EFERENCES

V. D ISCUSSION
In these seven applications the use of musical elements
made possible advanced music synchronization unattainableby typical music visualizers that for music synchronization
simply use the amplitude or spectrogram of audio signals.
Each application has a different way of using the elements
to control animation or physical devices (Table I), but thehierarchical beat structure (musical beats and bar lines) isalways used to enable people to feel the music synchronization. The musical beats are usually used to show regularchanges or motions. The bar lines (downbeats) are also
useful for adding accents at a higher level. This basic synchronization is important but is insufcient because it often
seems monotonous. The music structure (chorus sections andrepeated sections) is therefore important to change the wayof the music-sync control dynamically and temporally withina song. For example, most applications control the target in
the most intense and appealing way during chorus sections.
Some applications also make the target’s motion more interesting by linking different combinations of repeated sections
to different motions or changes. The chords (root note andchord type) are also used to change the mood of the control.

[1] M. Goto, K. Yoshii, H. Fujihara, M. Mauch, and T. Nakano, “Songle: A
web service for active music listening improved by user contributions,”in Proc. of ISMIR 2011, 2011, pp. 311–316.
[2] A. Klapuri and M. Davy, Eds., Signal Processing Methods for Music
Transcription. Springer, 2006.
[3] M. Casey, R. Veltkamp, M. Goto, M. Leman, C. Rhodes, andM. Slaney, “Content-based music information retrieval: Current directions and future challenges,” Proceedings of the IEEE, vol. 96, no. 4,pp. 668–696, 2008.
[4] M. Muller, D. Ellis, A. Klapuri, and G. Richard, “Signal processing
for music analysis,” IEEE J. of Selected Topics in Signal Processing,vol. 5, no. 6, pp. 1088–1110, 2011.
[5] M. Goto and Y. Muraoka, “A beat tracking system for acoustic signalsof music,” in Proc. of ACM Multimedia 94, 1994, pp. 365–372.
[6] J. Foote, M. Cooper, and A. Girgensohn, “Creating music videos using
automatic media analysis,” in Proc. of ACM Multimedia 2002, 2002,
pp. 553–560.
[7] F. Kurth, M. Müller, D. Damm, C. Fremerey, A. Ribbrock, andM. Clausen, “SyncPlayer — an advanced system for multimodal musicaccess,” in Proc. of ISMIR 2005, 2005, pp. 381–388.
[8] M. Goto, “A chorus-section detection method for musical audio signals
and its application to a music listening station,” IEEE Trans. on ASLP,
vol. 14, no. 5, pp. 1783–1794, 2006.
[9] M. Hamasaki, M. Goto, and T. Nakano, “Songrium: Browsing andlistening environment for music content creation community,” in Proc.
of SMC 2015, 2015, pp. 23–30.

88