Songle: A Web Service for
Active Music Listening
Improved by User Contributions
AIST (National Institute of Advanced Industrial Science and Technology)
Masataka Goto
Kazuyoshi Yoshii
Hiromasa Fujihara
Matthias Mauch
2011/10/26 ISMIR 2011
Tomoyasu Nakano
No. 1

Our Goal Enrich music listening experiencesby music-understanding technologies
based on signal processing
 Toward this goal, we have already developed

Active Music Listening Interfaces[Goto, 2003-]

 Active music listening• A way of listening to music through active interactions• You can take control of your own listening experience
No. 2

Active Music Listening Interface
 SmartMusicKIOSK (chorus-section detection)[Goto, 2003-06]• Access directly your favorite part of a song
• View a visual representation of its music structure

Similar (repeated) sections
Mapmc
usi

Chorussections
Repeated
sections“Jump to chorus”
buttonNo. 3

Active Music Listening Interface
 SmartMusicKIOSK (chorus-section detection)[Goto, 2003-06]• Access directly your favorite part of a song
• View a visual representation of its music structure

Similar (repeated) sections

dahtonsChorusahcilbuhpclua
sections
sregnniedsgnuafsoTheeeccafnraeethfRepeated
inclileylei
thvadlenhicrsaei
golsections
reseonhcet
apmci
Mus

“Jump to chorus”
buttonNo. 4

Songle (http://songle.jp)
 Web service for active music listening• Allow anonymous users to enjoy
any songs (MP3 files) available on the webby using active music listening interfaces

No. 5

Demonstration (Web Service)No. 6

Songle (http://songle.jp)
 Use automatic music-understanding technologies• Estimate four major types of music scene descriptions
Music structure(chorus / repeated sections)

Chords(root note and chord type)

Melody line

(F0 of the vocal melody)

Beat structure(musical beats and bar lines) A user can enjoy playing back a song while seeing

the visualization of the estimated descriptions
No. 7

Demonstration (Visualization)

No. 8

Demonstration (Visualization)

Songle facilitates deeper understanding of musicby visualizing music scene descriptions

No. 9

Songle with SmartMusicKIOSK Implement all functions of SmartMusicKIOSK• Jump and listen to the chorus with just a push of a button

Chorus

Chorus Chorus

Repeated sections

No. 10Demonstration (Active Music Listening)

No. 11

Demonstration (Active Music Listening)

Songle makes it easier for a user to finddesired parts of a piece

No. 12

Estimation Errors
 Automatic music-understanding technologies• Estimation errors are inevitable• Given the variety of musical pieces on the web
Music structure
ChordsMelody line
Beat structure Users of Songle might be disappointed

by its performance
No. 13Let’s Overcome Estimation Errors Together
 Songle enables anonymous users to contributeby correcting estimation errors
to improve the web service
 Efficient error correction interface (editor)• If a user finds an error while listening,
the user can easily correct the error
by selecting from a list of candidates orby providing an alternative description
No. 14

Demonstration (Error Correction)
 Music structure

No. 15

Demonstration (Error Correction)
 Beat structure

No. 16

Demonstration (Error Correction)
 Melody line

No. 17

Demonstration (Error Correction)
 Chords

No. 18Demonstration (Error Correction)
 Chords

The resulting corrections are then shared and used
to immediately improve the user experienceNo. 19Let’s Enrich Music Listening Experiences Songle is not just for collecting annotations,but for better user experiences• Users can simply enjoy active music listening
without correcting any errors!
• We understand that it is too difficult for some users to correct

 Users are not expected to correct all errors,

only some according to each user's interests

No. 20Songle as a Showcase
 Songle serves as a showcase• Demonstrate how people can benefit frommusic-understanding technologies• Understand their nature through user experiences“What kinds of sound mixture are difficult to handle?”
 Prevent overestimation of the technologies behind• The originally estimated values are visualized as trailswith different colors after user corrections• All correction histories are recorded and can be comparedNo. 21Songle as a Research Platform We hope Songle will serve as a research platform• Other researchers could exhibit results of
their own music-understanding modulesto jointly promote the popularization of our research field• The modules can be executed anywhere in the world
without sharing source and binary codes
Receive an audio file and send back the results via HTTP

• Interesting to visualize/compare/integratedifferences of music-understanding results• Let’s add other types of music scene descriptions
No. 22

Implementation
 Web server• Ruby on Rails, MySQL, Passenger, Apache Client user interface• ActionScript 3, Adobe Flex Compiler, JavaScript
 Music-understanding modules• Music structure: RefraiD [Goto, 2003-06]• Beat structure: HMM-based method [Mauch, 2010-11]
Supporting tempo changes, and 3/4, 4/4, and 6/8 meters• Melody line: PreFEst [Goto, 1998-2004] + VAD [Fujihara, 2006-10]• Chords: HMM-based method [Mauch, 2010-11]
14 types x 12 root notes, encouraging chord changes at bar lines
No. 23

Conclusion
 Songle: Web service for active music listening• 2009/06
Started the project and development• 2011/10/20
Launched the beta version• Designed to set into motion a positive spiral(1) Enable users to experiencea service based on music understanding
to let them better understand its performance(2) Users contribute to improved performance(3) The improved performance leads to a better user
experience, encouraging further use of the service at step (1)Game-based or crowdsourcing approaches often lack step (3)and depend on the feeling of fun or money
No. 24[Turnbull et al., ’07] [Mandel et al., ’07] [Law et al., ’07][Lee, ’10][Mandel et al., ’10]

Conclusion
 Songle as a social correction framework• Users gain a real sense of contributing fortheir own benefit and that of others• Users can be further motivated to contributeby seeing corrections made by other users
 Future work• Use corrected errors and machine learning techniques
to gradually improve music-understanding technologies
Better user experiences
Better technologies

No. 25

Let’s Enjoy and Collaborate!

http://songle.jp
English and Japanese versions are available.
You can register any song (MP3 file) on the web!
No. 26