AIST

supporting livelihoods!

Music-understanding technologies to realize the full potential of digital music

Proposing New Ways to Enjoy Music for
Supporting the Digital Content Industry

Engendering a "Content-Symbiotic Society" in which
content can be created and published without hesitation
Media Interaction Group, Information Technology Research Institute

Our life and society willchange in this way!
Music has been digitized and it is now possible to accessa huge number of musical pieces. Technologies by which
computers can automatically analyze and understandthis vast body of music will make daily appreciation ofmusic more enjoyable and enable anyone to easily create
music. This change could lead to a drastic change of theculture around music. AIST's music-understanding
technologies will open up new ways to enjoy music in thefuture and will provide a boost to content industries.

Making computers "understand" music
Masat aka Goto, a Pr ime Senior Researcher of theInformation Technology Research Institute, knows thatmusic-understanding technologies are necessary. He
explains: "When I was shopping for CDs to encounterunfamiliar but interesting music in my twenties, I would goto a trial-listening corner and repeatedly press the fastforward button to listen to chorus sections (“hook”) of asong. Since then, digital music has become the norm and,realizing that I would not be able to hear everything even if
I spent my whole life listening to music, I decided that it
would be helpful if computers could do the listening forme." From this point, he started on a quest to work out how
computers could be made to understand music in a way thatis useful for people.
When humans listen to music, they do things like hummingalong to the melody, clapping in time with the beat, andinstantly identifying the chorus. Although these things areeasy for humans, it is difficult to implement automatic
understanding of music by a computer if its mechanism isnot clear. Since music is one of the most complicatedsignals with multiple sounds being mixed together, existing
26

From AIST to the Innovative World

technologies were not enough to overcome the difficulties
of understanding music. Toward achieving automaticmusic-understanding technologies, Goto and his colleagues
have been researching and developing a wide range of coretechnologies for over 20 years and have become world
leaders in the field.
A web service "Songle" (launched in 2012) enables anyone
to easily try some of such technologies. Songle is distinct
from ordinary music players in that a user can view theentire structure of a song as a music map that is visualized
by using music-understanding technologies. A user canjump and listen to the chorus with just a push of a button,
which is useful not only for trial listening but also forcomparison between repeated choruses.

Aggregating the power of the crowd toimprove usefulness
Songle has applied music-understanding technologies toover 700,000 songs that have already been open to thepublic on the Internet, and their results can be enjoyed byeveryone in the form of music maps. There are some errors
in the music maps, but this is to be expected. The reality isthat AIST is conducting an important experiment in how tomake imperfect technologies useful for society. Songle isprovided with a function that enables anyone who can findand fix errors to contribute by making corrections. In thisway, Songle can make use of a synergy between thecapabilities of computers with their great processing power
but imperfect understanding and humans with their ability
to spot and fix errors. In other words, we are studying
systems of harmonious cooperation between computers andhumans that will be critical to society in the future.
Another web service "Songrium" (launched in 2013) uses
the automatic music-understanding results from Songle andenables a user to understand relationships between large
numbers of songs from a variety of viewpoints. InSongrium, a user can also skip ahead to the chorus and caneasily find songs with similar moods to their favorite songs,
videos in which people are singing and dancing along withthose favorite songs, and the like. In an era in which new
original songs are first released on the Internet, such newImpact in the following fields!

Songle▶ ‌The results of automatic
understanding of thecontent of a song on theInternet are visualized
as a music map. While
viewing the structure ofthe song, the user may
listen only to a rousing
chorus or may compare
repeated sections.
(From‌an‌AIST‌press‌release‌of‌August‌29,‌2012)

Community life◦Services◦ ‌Sports and leisure◦Education

Industry◦IT and Telecommunications◦Electronics◦Services◦ ‌Software and content

http://songle.jp
Music content (MP3,Piapro, SoundCloud, Automaticmusic-understanding
Niconico, YouTube)

Active Music Listening ServiceMusic structuretechnologies
Music playback Internet (web)

Users (anyone canuse for free)Music visualization
Chorus search

Using web browsers

Chorussections

Chords“Music Map”
Visualized musiccontent

Repeated sections
Root note and chord type

Melody line

Tone of the vocal melody

Beat structureMusical beats and bar lines

Correction of estimation errors

ways to enjoy music are continuing to open up.

http://songrium.jp

Songrium

Researching singing synthesis
technology for creating music
The advance of technology is bringing about a new cultureof music. Singing synthesis technology, which has been
attracting attention since 2007, has produced, for the firsttime in history, a culture that can positively enjoy songs
with synthesized voices as main vocals. This innovative
culture is an expression of Japan's strengths, and AIST is
pursuing research to take this culture further forward.
O ne s uch si ng i ng sy nt he sis t e ch nolog y is ca l le d
VocaListener ("Bokarisu"). This technology enables thesynthesis of singing voices that imitate the way a usersings. The user need only input a sample of their singing
into VocaListener, which then controls commerciallyavailable singing synthesis software with various voicetimbres to naturally synthesize voices that imitate thesinging.
I n t h is era of d ig it al music, music-u nderst a ndi ng
technology is supporting access to the huge body of songson the Internet, and singing synthesis technology isexpanding the scope for enjoyment in creating music.
These trends will not be limited to music but will spread toother forms of content, such as videos.
Goto adds that "There is a problem of whether, as thedigitalization of content produces a society in whichnothing is forgotten, the future might be buried under thehuge and continually growing body of content from thepast. For listeners, selecting music is getting more difficult,and for creators, their creations can easily just disappear
into obscurity. In addition, as more and more similar
content appears, the risk that a person's creations will be
criticized for being too similar to other pieces increases,
and there is a danger that we will end up with a society inwhich people hesitate to create and publish their owncontent. By continuing with research and development
powered by AIST’s core technologies, we are aiming for a"Content-Symbiotic Society" in which everyone can enjoy
appreciating and creating music without hesitation while
paying due respect to older content."▲ Songrium visualizes various musical relations between
music videos on video sharing websites. The user can keep
encountering new songs and review a history of old songs.
(From‌an‌AIST‌press‌release‌of‌August‌27,‌2013)

Toward making singing synthesis more natural
Singing synthesis software, such as Hatsune Miku
created by Crypton Future Media, uses VOCALOID, asinging synthesis technology developed by Yamaha. In
the past it was difficult to produce natural-sounding
singing with this technology but AIST overcame thisproblem with "VocaListener". Yamaha started to sell acommercial version of VocaListener in 2012.
(From‌a‌Yamaha‌press‌release‌of‌September‌18,‌2012)

Given the user’s singing voice and the lyrics, it can synthesize the singing voice mimicking the user.
Example of the user’s singing voice

Tone of the voice

Lyrics “Tachidomarutoki, Analysis
mata futo furikaeru”

Singing synthesis software ASinging synthesis software B
Several singingsynthesis softwares

Several sound source(singer’s voice)Time

Loudness of the voice

Guessing parameters

Time

Natural singing voice like a human
Synthesis

Synthesis

Even by switching either a singing synthesis software , or a sound source (singer’s voice), it
can always synthesize the singing voice in the same way (tone and loudness of the voice).

AIST supporting livelihoods!

27