Songle Sync: A Large-Scale Web-based Platform forControlling Various Devices in Synchronization with MusicJun Kato, Masa Ogata, Takahiro Inoue, Masataka Goto
ACM Multimedia 2018 Oct. 25, Poster Session 5

National Institute of Advanced Industrial Science and Technology (AIST)

Songle Sync drives a variety of hundreds of devices synchronized with a musical piece on the web♬♫
♪♬♫
♬
♪
♫♪
Music♪
♬♫
on the♬Web♫♬♪
♬♫♪ ♬♫♪

♬♬♬

Large-scale web-based platform Songle Sync

Semantic information extracted withmusic-understanding technologies♬♫♪♬♫♪
♬♫♪♬♫♪

Playback status and timeline sharedon the autonomous control architecture♪

♬♪
♬Smartphones

Music drives various devices forsynchronized multimedia performanceTabletsRobots)
DOYB(eciveDnw
Bring-Your-O!
senohptramsr
experience foMusic-Driven Multimedia Performance at Scale
1ecieplaicsumaeChoos
and animation style

Prior work:by a single device
Automatic music analysis has enabledmusic-driven multimedia performances

PC Laptops

IoT devices

3

Generateyour own
QR code!

deiznorhcnysehtyjonEperformance!

detarenegehter
Shasdnierfhitwedoc
QR

2Our work is the first attempt to provide a platform on which music-driven performances involvea variety of hundreds of devices.

Dynamic Hardware Setup

Scalable Control of Devices

Stable Control of Devices

Various Internet-connected
smartphones can join the session

synchronizes >1000 devices
without significant latency nor jitter

works stably under challenging
networking environments

Heterogeneous Hardware Setup

Open Platform with Development Kit

Lighting devicesplayer.on(”play”, listener);player.on(”beatEnter”, listener);
Off-the-shelf mobile devices and computers

Wearable devices

Robots and other actuated devices

Event-driven APIs for easily controlling devices with JavaScript
Code for one device can synchronize hundreds of devices
No need to worry about networking and synchronization

Various JavaScript-driven devices (e.g., Raspberry Pi and Intel Edison)can be controlled since standard web technologies are utilized

Novel “Autonomous Control Architecture”
masterConventional “Always-On Architecture”
Widely used in live performanceswith dedicated hardware/smartphone appsMaster node emits a command at every event
Slave nodes react to the command
Per-event communication ➡ latency and jitter

Example programs and interactive tutorials

Evaluations
1801601401201008060

Master node chooses a musical piece
Songle Sync distributes timings ofmusical elements (e.g., beats and chords)
Playing status is shared periodically...

40200-20 [ms] (latency)

Deployments in the Wild

Performance
Comparison
Always-On:
0-180[ms] delays
Autonomous:-20-0[ms] delays<100[ms] jitter
observed by a960fps camera

MUSIC HACK DAY Tokyo

Demo room experimentwith >110 heterogeneous
hardware setup
Live performance with>275 synchronizedsmartphones
Recent event synchronized>1200 devices

The era of “Internet of Musical Things (IoMT)” has come!

More details at
http://api.songle.jp/sync

masterEvery node knows event timings
before the multimedia performance
No per-event communication ➡
theoretically no latency nor jitter
NTP-like protocol to sync clocks

Dev. Kit Usability Test
2-day hackathon with 24
university students