Keynote & Invited Talk

IUI 2018, March 7–11, 2018, Tokyo, JapanIntelligent Music Interfaces
Masataka Goto
National Institute of Advanced Industrial Science and Technology (AIST), Japanm.goto@aist.go.jp
ABSTRACT

based multimedia development framework, “SongleWidget” (http://widget.songle.jp) [5], that enables musicsynchronized control of computer-graphics animation androbots. It makes it easy to develop web-based applications
with rigid music synchronization by leveraging musicunderstanding technologies. Furthermore, by using webmining and music-understanding technologies, a webservice for large-scale music browsing, “Songrium”(http://songrium.jp) [6], provides various bird’s-eye views
on more than 780,000 music video clips on video-sharing
services.

Automatic music-understanding technologies (automatic
analysis of music signals) make possible the creation of
intelligent music interfaces that enrich music experiences
and open up new ways of listening to music. In the past, it
was common to listen to music in a somewhat passive
manner; in the future, people will be able to enjoy music ina more active manner by using music technologies.
Listening to music through active interactions is called“active music listening.”
In this keynote speech I first introduce active music
listening interfaces [1] demonstrating how end users canbenefit from music-understanding technologies based on
signal processing and/or machine learning. Musicunderstanding technologies can automatically estimate
music scene descriptions [2] in musical audio signals:
descriptions such as melody and bass lines, beat structure(beat and bar), and music structure (chorus section). By
using the estimated music structure, for example, the active
music listening interface “SmartMusicKIOSK” [3] enables
people to access their favorite part of a song directly(skipping other parts) while viewing a visual representationof the song’s structure. Other interfaces enable people toalso customize (personalize) music by changing the volume
or timbre of instrument sounds in existing music recordings
and to browse a large music collection to encounterinteresting musical pieces or artists.

In the future, further advances in music-understanding
technologies and music interfaces based on them will makeinteraction between people and music even more active andenriching.
Author Keywords

Music interface; music understanding; music analysis;
active music listening; web service; music information
research
BIOGRAPHY

I then introduce our recent challenge of deploying suchresearch-level music interfaces as web services open to thepublic. For example, a web service for active music
listening, “Songle” (http://songle.jp) [4], has analyzed morethan 1,100,000 songs on the web and augments people’s
understanding of music by visualizing music scene
descriptions estimated automatically. Since the current
accuracy of automatic music understanding is less than thatof human music understanding, Songle features acrowdsourcing interface that enables users to collectively
correct analysis errors. Songle is used to provide a web-Masataka Goto received the Doctor of Engineering degreefrom Waseda University in 1998. He is currently a Prime
Senior Researcher at the National Institute of Advanced
Industrial Science and Technology (AIST). In 1992 he wasone of the first to start working on automatic music
understanding and has since been at the forefront ofresearch in music technologies and music interfaces basedon those technologies. Over the past 25 years he has
published more than 250 papers in refereed journals andinternational conferences and has received 46 awards,
including several best paper awards, best presentation
awards, the Tenth Japan Academy Medal, and the Tenth
JSPS PRIZE.

Permission to make digital or hard copies of part or all of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. Copyrights for thirdparty components of this work must be honored. For all other uses, contact
the owner/author(s). Copyright is held by the author/owner(s).
IUI 2018, March 7–11, 2018, Tokyo, Japan.
ACM ISBN 978-1-4503-4945-1/18/03.
DOI: https://doi.org/10.1145/3172944.3176184

3

Keynote & Invited Talk

IUI 2018, March 7–11, 2018, Tokyo, JapanHe has served as a committee member of over 110
scientific societies and conferences, including the General
Chair of the 10th and 15th International Society for Music
Information Retrieval Conferences (ISMIR 2009 and 2014).
As the Research Director he began a 5-year research project(OngaCREST Project) [7] in 2011 and follow-on 5-year
research project (OngaACCEL Project) in 2016, both ofwhich have focused on music technologies and been fundedby the Japan Science and Technology Agency(CREST/ACCEL, JST).

Speech Communication (ISCA Journal), Vol.43, No.4,pp.311-329, September 2004.
3. Masataka Goto: A Chorus-Section Detection Methodfor Musical Audio Signals and Its Application to aMusic Listening Station, IEEE Transactions on Audio,
Speech, and Language Processing, Vol.14, No.5,pp.1783-1794, September 2006.
4. Masataka Goto, Kazuyoshi Yoshii, Hiromasa Fujihara,
Matthias Mauch, and Tomoyasu Nakano: Songle: A
Web Service for Active Music Listening Improved byUser Contributions, Proceedings of the 12th
International Society for Music Information Retrieval
Conference (ISMIR 2011), pp.311-316, October 2011.

https://staff.aist.go.jp/m.goto/
ACKNOWLEDGMENTS

I thank (in alphabetical order) Hiromasa Fujihara, Satoru
Fukayama, Masahiro Hamasaki, Takahiro Inoue, Keisuke
Ishida, Yuta Kawasaki, Matthias Mauch, Tomoyasu
Nakano, Kosetsu Tsukuda, and Kazuyoshi Yoshii, whohave been working with me to build music interfaces andweb services. This work was supported in part by JST
ACCEL Grant Number JPMJAC1602, Japan.

5. Masataka Goto, Kazuyoshi Yoshii, and Tomoyasu
Nakano: Songle Widget: Making Animation andPhysical Devices Synchronized with Music Videos onthe Web, Proceedings of the IEEE International
Symposium on Multimedia (IEEE ISM 2015), pp.85-88,December 2015.
6. Masahiro Hamasaki, Masataka Goto, and Tomoyasu
Nakano: Songrium: A Music Browsing Assistance
Service with Interactive Visualization and Explorationof a Web of Music, Proceedings of the 23rd
International World Wide Web Conference (WWW2014), pp.523-528, April 2014.

REFERENCES

1. Masataka Goto: Active Music Listening Interfaces
Based on Signal Processing, Proceedings of the 2007
IEEE International Conference on Acoustics, Speech,and Signal Processing (IEEE ICASSP 2007), pp.IV1441-1444, April 2007.

7. Masataka Goto: Frontiers of Music Information
Research Based on Signal Processing, Proceedings ofthe 12th IEEE International Conference on Signal
Processing (IEEE ICSP 2014), pp.7-14, October 2014.

2. Masataka Goto: A Real-time Music-scene-description
System: Predominant-F0 Estimation for Detecting
Melody and Bass Lines in Real-world Audio Signals,4