ICSP2014 Proceedings

FRONTIERS OF MUSIC INFORMATION RESEARCHBASED ON SIGNAL PROCESSING

Masataka Gata
National Institute of Advanced Industrial Science and Technology (AIST), Japanm.gata [at] aist.ga.jp

been attracting attention since 2007, and the most surprising change
in this regard is the birth in Japan of the world's first culture that ac­

ABSTRACT

How will music infonnation research open up the future to new waysof enjoying music? Today it is the fate of all music in the world tobe digitized, and the era in which we can listen to a large selectionof our favorite music anytime and anywhere has arrived. In the past
we generally listened to music passively, but in the future we will beable to enjoy music in a more active manner. In this keynote paper,
I will discuss how music technologies will open up new possibilities
in both music appreciation and music creation.

tively e/ljoys songs with synthesized singing voices as main vocals,

thereby breaking down the long-cherished view that listening to a
non-human singing voice is worthless. This is a feat that could nothave been imagined prior to 2007, when the singing-voice synthe­
sis software Hatsune Miku [3, 4] based on VOCALOlD [5] singing
synthesis technology first went on sale. Further advances in mu­
sic information research may break down other entrenched attitudes
about music listening and give birth to another new culture of music
appreciation.
In the light of this increasing importance of music infonnation
research, this paper presents my ideas on hypotheses and grand chal­lenges in the hope that music information research will make more
contributions to academia and society while opening up new pos­sibilities in the future. I have already discussed five challenges in
my article "Grand Challenges in Music Information Research" [6].
Here, in Section 4 I purposely take up more controversial topics thathave much room for debate. Although some of them are not very
technically feasible at this point in time, I hope that our researchcommunity will foster new ideas by freeing ourselves from current
attitudes about music.

Index Terms- Music infonnation research, music technologies
1. INTRODUCTION

Music information research has gained in importance academically,
industrially, socially, and culturally, thereby attracting an increasing
amount of interest. The history of music information research is along one, as attempts to apply computers to the automatic compo­
sition of music began not long after the invention of the computer,the best example being the IIIiac Suite in 1957. Research achieve­
ments in this field have spread widely through society, such as musicstreaming services and the sound synthesizers now essential to music
production.
In addition, many people have, especially since the start of thiscentury, come to realize that all music will eventually be digitized
and will be created, delivered, retrieved, shared, and disseminatedin that fonn. As a result, the importance of music information re­search has become widely recognized and new researchers through­out the world have begun to enter this field. Indeed, with the growing
awareness that the already huge and ever increasing amount of music
content must be dealt with in a better way, research on music infor­
mation retrieval, music recommendation, and new music interfaces
has become active.
Another reason for this growing interest in music information
research from an academic viewpoint is the difficulty of understand­ing audio signals that convey content by forming a temporal struc­ture consisting of multiple and interrelated sounds. Automatic un­derstanding of such signals is a fundamental unresolved problem.
Other unresolved but intriguing problems that have yet to be touched
provide a treasure-trove of future research themes. The number ofinternational conferences and workshops presenting achievements inmusic information research continues to increase, and given the im­portance and future possibilities of this field, research projects withsizable budgets have been launched one after another in Europe, theUnited States, and Japan.
Along with the spread of digital music content in society, indus­
trial applications of music technologies have already expanded invarious forms and made contributions to society. Singing informa­tion processing [1,2] typified by the synthesis of singing voices has

978-1-4799-2186-7114/$31.00 ©2014 IEEE

2. FUTURE POSSIBILITIES WITH MUSICINFORMATION RESEARCHMusic information research holds a variety of possibilities for the
future. Each and every researcher carves out a future that he or she
believes in, which results in various research approaches that could
be classified, for example, from the following viewpoints:•"Analysis and understanding" versus "generation and cre­ation"•"Signal processing" versus "symbol processing"•"Real-time and interactive" versus "non-real-time and noninteractive"•"Creating a system" versus "learning about people"•"Dealing with one song" versus "dealing with a set of songs"•"Dealing with the song itself' versus "dealing with metadata
about a song"•"Communication by perfonning music" versus "meta-levelcommunication while listening to music"•"Professional-oriented" versus "amateur-oriented"•"Automatic" versus "manual or semiautomatic"

In thinking about the future direction of this research field, in thispaper I focus on the following two important approaches related to
the last of the above viewpoints.
7(1) Research on augmenting human abilities (achieving "musicalability support")
In addition to supporting training processes enabling people
to develop their musical abilities, this approach will include
support for temporarily augmenting (enhancing) human abil­ities. For example, a person who does not have the ability tocompose could produce a musical piece by using a composi­tion support system, or a person who is not good at singing
could produce a main vocal by using a singing synthesis sys­tem. Even electronic musical instruments (sound synthesiz­ers) could also be considered as support for augmenting themusical ability of a person who cannot play a certain musicalinstrument. I will discuss "musical ability support" in moredetail in Section 3.

(a) Training: Improve human ability itself(b) Support with an immediate effect: Temporarily improve hu­
man ability through the power of technology(b-I) Make anyone a musician: Empower a person lacking
musical abilities(b-2) Give a musician outstanding ability: Make a musician
with ability even better"Training" here corresponds to providing people with the meansof continually or permanently improving their own abilities throughtechnology. Examples of this type of support include vocal train­
ing [8-10] and performance training [11-17]. Although human abil­
ity can be improved through steady effort and practice even without
technology-based support, this research aims to improve human abil­
ity in a more efficient and appropriate manner.
"Support with an immediate effect," in contrast, corresponds tothe enhancing of some kind of musical ability in a person for thetime that technology-based support is being provided even if that
person's ability itself will not be fundamentally changed. In otherwords, this type of support aims to empower people only for the timethat the technology in question is being used - there is no need to
wait for an improvement in human ability. Depending on who is
being empowered and to what extent, I divide this type of support
into "Make anyone a musician" and "Give a musician outstanding
ability."
In "Make anyone a musician," it is assumed that the user is an
ordinary person without musical training. Research in this area aims
to use the power of technology to make easy what is usually difficult
for that person. Efforts in this regard include composition support
in a way that reflects the intent of the user in some form [7, 1821], performance support that enables the user to perform as desired
through operations on a computer even if that person cannot play amusical instrument [12,22-25], and vocal support that synthesizes
or processes singing voices [4, 5,26,27]. This approach targets eventrained musicians who lack certain abilities and would like to receive
support for them.
In "Give a musician outstanding ability," however, the user isassumed to be a trained musician and is enabled to give musicalexpressions that are difficult for an ordinary musician. A musician
whose abilities have been enhanced through computer support could
be expected to seek out new forms of expression. If we were tosimply think of this as "support for outstanding performances," theneven a high-speed mechanical performance by synthesizers would
be this kind of support, as would physically augmenting the bodycontrol of a musician by using some devices.
In reality, there could be a "Training" effect when one is re­ceiving "Support with an immediate effect," and there could be anintermediate support between making anyone a musician and givinga musician outstanding ability. So these kinds of support can also
take on various overlapping forms.
The enhancement technologies discussed above were focusedon augmenting human abilities in relation to "music generation"
as in music production and music performance. In the follow­
ing, however, I introduce some examples of technologies enhanc­
ing human abilities in relation to "analysis and understanding" as inmusic appreciation: "Augmented Music-Understanding Interfaces"(Section 3.1) and the "OngaCREST Project: Similarity-Aware In­formation Environment" (Section 3.2), both of which my colleagues
and I have been working on. Various researchers are also conducting
studies on other means of augmenting human abilities.

(2) Research on automating musicians' abilities (achieving a"virtual musician")This approach attempts to realize on a computer various abil­ities possessed by human musicians. Research themes tar­geted by this approach include automatic composition, au­tomatic arrangement, automatic generation of lyrics, auto­
matic performance (automatic playback of music files, auto­
matic improvisation, and performance rendering), automatic
accompaniment, automatic recognition of sheet music (opti­
cal music recognition), and automatic transcription. This ap­
proach may achieve human abilities by imitating them or may
achieve functions that simply exceed human abilities. In theautomatic control of musical instruments, for example, ren­derings that cannot be performed by humans are frequently
used regardless of actual human ability. On the other hand,automatic performance can also refer to efforts at automat­
ically generating an improvised performance in the mannerof a jam session or at performing a certain musical score ina human-like manner. In Section 4 I will continue this dis­cussion and elaborate on the future direction of the virtual
musician concept.
Approaches (1) and (2) above are not independent and just dif­
fer in their focus. If an ability achieved by a virtual musician is usedwith the aim of supporting a person without that ability, it then be­comes a technology in the category of "musical ability support" inapproach (1). For example, the automatic composition system Or­
pheus [7] that enables a user to automatically compose a song byentering lyrics and selecting provided options would obviously beclassified as approach (2) but if viewed as a technology for augment­
ing the ability of a human to compose music would be classified as
approach (1).
It is interesting to investigate how much contribution a person
has to make in order to feel that the process is not totally automaticand that he or she is simply being supported. What is necessary for a
person using music technologies to be able to think "I am engaging
in self-expression by using my own abilities that have simply been
augmented"? If it were simply a matter of pushing a button, it would
be difficult to think of the results obtained as self-expression no mat­ter how much the technology might be referred to as content-creation
support.
3. AUGMENTING HUMAN ABILITIESAs described in "Research on augmenting human abilities' inSection 2, a variety of approaches have come to be taken with theaim of augmenting abilities and overcoming limitations through thepower of technology. I think that the enhancement of human abilities
can take several directions, which I classify as follows:83.1. Augmented Music-Understanding Interfaces

song by equalizing in units of musical instruments. Dru­mix is a music playback interface that allows the user to edit
the drum part of a song in real time during playback just as
if another drummer were performing a different drum pat­tern. These interfaces augment the ability of the user to listen
for and understand different parts corresponding to individual
musical instruments.

I proposed a research approach called Augmented Music­
Understanding Interfaces [28,29] that aims to improve one's abil­
ity to listen to music. These interfaces are examples of the "Makeanyone a musician" approach described in this section.
The goal of this approach is to enrich music listening experi­
ences by deepening each person's understanding of music. Music
listening experiences depend on music-understanding abilities. Al­
though the music-composing/performing abilities of musicians areoften discussed, the music-understanding abilities of casual listeners
have not been discussed or studied much. It is difficult, for exam­
ple, to define music-understanding abilities and express the resultsof understanding. In addition, it is difficult to know how others un­derstand music. Music listening is usually an individual experience,and it is impossible to directly observe how others understand ele­ments in music. Similarly, one often does not notice what one does
not understand when listening to music. Methods that listeners canuse to better understand music or improve their ability to understandmusic have not yet been established and will have to be discovered.

For example, a person who is usually unaware of the drum
pattern or timber of drum sounds in a song can use Drumix
to edit those drum characteristics, which should help that per­son distinguish between bass-drum sounds and snare-drumsounds and learn how drum patterns can have a great effecton the feeling conveyed by a song. We believe that interfaces
for such casual experience of derivative creation can be as­sociated with content-creation support technologies and con­tribute to an "age of billions of creators."
These interfaces were originally developed as Active Music Lis­
tening Interfaces [35] enabling people to enjoy music in active ways

but were later found to be good examples of Augmented Music­
Understanding Interfaces. The SmartMusicKIOSK interface canbe experienced as part of the active music listening service Songle(http://songle.jp) [36,37].

3.1.1. Examples of Augmented Music-Understanding Interfaces

My colleagues and I have therefore pursued a research approach ofbuilding Augmented Music-Understanding Interfaces [28, 29] thatfacilitate deeper understanding of music by using automatic music­
understanding technologies based on signal processing. The follow­
ing are examples based on this approach.
•3.1.2. Discussion

The following functions provided by the interfaces described above
are important in augmenting music understanding:•

SmartMusicKIOSK: Music listening station with a chorus­
search function [30,31]
SmartMusicKlOSK is a music playback interface that visu­

alizes the entire song structure consisting of chorus sections
and repeated sections to enable within-song browsing of pop­
ular music and trial listening of songs. It augments the abilityof the user to understand music structure and musician intent.

LyricSynchronizer:

Automatic synchronization between mu­

sic and lyrics [32]LyricSynchronizer is a user interface that displays scrolling

lyrics with the phrase currently being sung highlighted in amanner similar to that used in karaoke machines. It augments
the ability of the user to understand the lyrics of a song.

Synchronization of information display with music playback•

Provision of interactive interfaces for controlling music playback and musical elements

3.1.3. Toward MUSic-Understanding Ability Training Interfaces

The above Augmented Music-Understanding Interfaces can, fromthe viewpoint of music appreciation, be classified as a "Make any­
one a musician" approach to providing "support for temporarily aug­
menting the ability to listen to music" in Section 3. The next step
will be to develop interfaces that permanently improve one's ability
to understand music, as it is improved in the "Training" described in
Section 3. Although there are many ways to improve one's ability
to understand a foreign language, such as through language schools
and training materials, there are virtually no systematic means ofimproving music-understanding abilities. Most music schools andtraining materials including music-dictation training are intended formusicians and creators rather than casual listeners.
I therefore proposed the research approach "Music­
Understanding Ability Training Interfaces" [29] as an important

For example, LyricSynchronizer enables a person who usu­
ally listens to songs without concern for the lyrics to viewa song's lyrics synchronized with playback or to click on aword that appears interesting and listen from that word. In
this way, the user pays attention to a song's lyrics while lis­tening to it, and this may provide the user with a better under­
standing of the message conveyed by the lyrics.
••

Visualization of music content plays an important role in augment­ing (temporarily supporting) people's understanding of music be­cause understanding is deepened through seeing. And music touch­up (personalization or customization by making small changes toelements in existing music) [35], like those made using Drumix,
also helps music understanding because understanding is deepened
through editing. It lets users naturally observe why music is com­posed and arranged in a certain way while casually enjoying thecontent modification.
We also found that interfaces able to make users aware of what
they tend to not notice are useful. User interaction enhances immer­
sive music listening experiences and promotes a deep appreciationof music.

It can, for example, provide a person who usually listens tomusic without any concern for music structure an opportu­
nity to listen to music while being aware of that structure. In
addition, by enabling the user to listen to the chorus sectionsof a song in succession, it lets the user obtain a more accurate
understanding of how lyrics and arrangement change for each
repetition of the chorus (as a reflection of musician intent).
•

Visualization of a song's content (musical elements)

INTER: Musical instrument equalizer for music audio sig­

nals [33]; Drumix: Music playback interface with a real-timedrum-part editing function [34]
INTER is a music playback interface that allows the user toadjust the volume of each track in a song corresponding to adifferent musical instrument. This enables remixing of the9Building an information environment for specialists and general users

However, human ability with regard to similarity is limited.
Judging similarity between two things in front of one's eyes is usu­
al1y a simple task, but the speed of judging similarity is limited searching for similar content among a mil1ion items of content can­not be done for al1 practical purposes. Moreover, while humans areable to make accurate judgments based on past experience, theirability is limited in judging "commonness" as in determining theprobability that some event out of al1 others wi11 occur. When an
uncommon event happens to be frequently observed recently, for ex­
ample, people tend to wrongly assume that it is likely to occur. And
when a frequent event happens to not be encountered, people tend towrongly assume that it is rare. Consequently, with the coming of an"age of bil1ions of creators" in which anyone can enjoy creating andsharing works, the monotonic increase in content means that there isa growing risk that one's work wil1 be denounced as being similar tosomeone else's. This situation could lead to a society that makes it
difficult for people to create and share content without worry.

who want to know "What is similar here?" and "How often does this occur?"
Information environment that promotes an awarenessof similarity among a huge amount of media content

Similarity·aware Information Environme

Enhancement of human ability

Can understand the "similarity" and "commonness"within a huge amount of content

O.. Content ecosystem technologies

Content-creation and content-appreciation
supporltechnologiesf',�

Create and use media content in a richer, deeper way

Content-Symbiotic Society- Anyone can create and share content without worry- Anyone can casually engage in media content creation- Anyone can actively encounter and appreciate content

New Value
Content culture that respects past content
and emphasizes emotional experiences

Main target: Music (musical audio signals, etc.) andMusic-related video (music video clips, dance video clips, etc.)
Fig. 1. Overview of OngaCREST Project.

future direction towards enabling a greater number of people toenjoy music in more depth from more diverse views. The more
people there are who understand how delicious good food can be,
the more likely it is that delicious food will be prepared. In the sameway, the more people there are who understand the attraction andmagic of music, the more likely it is that attractive music will becreated. Augmenting and improving music-understanding ability isimportant to that end.

3.2.2. information environment enabling people to use similarity
and commonness

In light of the above, we started the OngaCREST Project to build anenvironment in which specialists and general users alike can know
the answers to the questions "What is similar here?" and "Howoften does this occur?" Here we aim to make it possible for peo­
ple to continue creating and sharing content without worry. Fur­thermore, we want to make it easy for anyone to enjoy the content
creation process by developing content-creation support technology
enabling "high commonness" elements (such as chord progressions
and conventional genre-dependent practices) to be used as knowl­edge common to mankind. We also want to promote a proactive
approach to encountering and appreciating content by developing
content-appreciation support technology that enables people to en­
counter new content in ways based on its similarity to other content.
We hope to contribute to the creation of a culture that can mu­
tual1y coexist with past content while paying appropriate respect toit. This will become possible by supporting a new music culturethat enables creators to take delight in finding their content being
reused in much the same way that researchers take delight in finding
their articles being cited. We feel that the value of content cannot
be measured by the extent to which it is not similar to other content- pursuing originality at al1 costs in content does not necessarily
bring joy to people. Fundamental1y, content has value by inducing
an emotional and joyous response in people. We would like to makeit a matter of common sense that content with emotional appeal andhigh-quality form has value. In fact, we would like to see conditions
in which it is exactly the referring to many works that gives content
its value, similar to the situation with academic papers. Through thisapproach, we aim to create a content culture that emphasizes emo­
tional1y touching experiences.
In the digital content society, future content is apt to be over­whelmed by the huge amount of ever-increasing past content but
without being forgotten. The OngaCREST Project takes up thechal1enge of creating a content-symbiotic society capable of rich,sustainable development in a "cannot-be-forgotten society" brought
about through digitization. Our aim here is to enable people to feelthe symbiosis between past and future content and to create a so­ciety that can enjoy a huge amount of content through a symbiosis
between people and content.
As described above, the OngaCREST Project can be posi­tioned as an endeavor to enhance and support human ability
in jUdging similarity and creating and appreciating content, al1
through the power of technology. Our achievements to date in

3.2. OngaCREST Project: Similarity-Aware Information Envi­ronment

Since judging music similarity is an essential ability if one is to un­derstand music, I introduce (as an example of the "Support with
an immediate effect" approach described in Section 3), a five-year
research project, named OngaCREST, intended to provide various
types of support for music creation and appreciation based on the im­
plementation and enhancement of this similarity jUdging ability. The
OngaCREST Project is official1y entitled "Building a Similarity­aware information Environment for a Content-Symbiotic Society,"

and is now in progress as a fiscal-year-2011 selected research project(Research Director: Masataka Goto) in the research area of "Cre­ation of Human-Harmonized information Technology for Convivial
Society" of the CREST (Core Research for Evolutional Science andTechnology) funding program provided by the JST (Japan Scienceand Technology Agency).
As shown by the project overview in Figure 1, this project aims
to build a similarity-aware information environment in which peo­
ple are able to know similarities among vast amounts of media con­tent so as to foster a content culture that respects past content andgives greater importance to emotional1y touching experiences. Fur­thermore, by developing technologies that can open up future pos­sibilities in content creation and appreciation, we aim to promote asociety in which anyone can engage in content creation and appre­
ciation. The main types of media content targeted in this project aremusic and music-related videos.
3.2. i. Limits of human ability in judging similarity

The amount of digital content that can be accessed by people has
been increasing and wil1 continue to do so in the future. This is de­sirable, but unfortunately makes it easier for the work of content cre­ators to become buried within a huge amount of content and makes
it harder for viewers and listeners to select content. Furthermore,since the amount of similar content is also monotonical1y increas­ing, creators wil1 be more concerned that their content might invite
unwarranted suspicion of plagiarism. Al1 kinds of works are influ­
enced by existing content and it is difficult to avoid the unconscious
creation of content partly similar in some way to prior content.
10

words "automatic generation of music," what would we think if themechanism of such automatic generation was actually a "virtual col­laboration" between people and computers based on N-th generationof derivative creation from a huge number of Consumer Generated
Media (CGM) works posted by people? In other words, how would
we react if the number of songs created by a huge number of users
within a CGM culture were to reach a massive number and new at­
tractive songs came to be automatically generated by computer on
the basis of the results of automatic music understanding targeting
those songs?

relation to the content-appreciation support technology describedabove include the web-based active music listening service Songle(http://songle.jp) [36, 37] and music browsing assistance service
Songrium (http://songrium.jp) [38,39].
4. WILL COMPUTERS CREATE AND APPRECIATEMUSIC IN THE FUTURE?

The virtual musician concept presented in Section 2 under the "Re­search on automating musicians' abilities" heading will cover both
"automatic creation" and "automatic appreciation." In the follow­
ing, I discuss the possibility of a future in which computers can au­
tomatically create and appreciate music. At this point in time, this isconsidered a grand challenge that may be somewhat divorced fromreality, if not science fiction.

4.2. Automatic appreciation

The above discussion assumes the audience, at least, to be humanbeings. However, could computers be an audience as well? Let'sconsider a future in which "automatic appreciation" is a possibility.
At present, the research on automatic music understanding bycomputer is often concerned with describing musical elements andphenomenon (music scene descriptions [40, 41]) expressed by mu­
sical audio signals, classifying music by genre, and tagging pre­trained labels associated with emotions and other characteristics.
Such research has produced, for example, the MusicCommentatorsystem [42] that automatically generates comments on a musicalpiece by using machine learning of the relationship between music
and time-synchronized user comments made on the Niconico video­
sharing service. Nevertheless, technology that enables a computer
to "appreciate" music in a form that includes preferences and value
judgments like a human being has not yet been realized. In short, thefeasibility of doing so with current technology is still low, but what
would we as human beings think if this became possible?
After creating a musical piece, a human being usually expects
other people to listen to it. The creator of a work would of course
place value on the act of creating a musical piece regardless ofwhether someone else listens to it but in general would hope that
another human being catches the messages or feelings expressed in
the piece. The possibility is therefore high that the creator of a mu­
sical piece would not place much value on it if his or her musicalpiece were listened to by a computer and not people. On the other
hand, we cannot rule out the possibility that this traditional attitudeof "there is no value in having one's music listened to by a non­human audience" will someday be made obsolete.
Furthermore, in a future in which both automatic creation andautomatic appreciation have become technically possible, it is con­ceivable that computers will function as both creators and appreci­
ators of music. If it cannot be assumed that only people appreciatemusic, human hearing organs and human power of understanding
will no longer limit music expression and the possibility arises thatmusical expression will be transformed, perhaps to forms with super­
fast tempos or extremely complex acoustical expressions. This isinteresting from a science-fiction perspective, but in reality a worldin which people create and appreciate music should continue evenin a future with computers creating and appreciating music. As has
happened in the past, new technologies will open up new musicalexpressions, but those expressions will fall into a range that can be
accepted and enjoyed by human beings.

4.1. Automatic creation

First consider "automatic creation." One of many future possibil­
ities is that it might be possible to totally automate the generationof musical pieces and even lyrics through the power of technology
and that this automation will give birth to a culture in which people
actively enjoy such music. Although many researchers have alreadyresearched automatic composition, I myself could not stop feelingthat songs I usually listen to should be composed by human beings.
However, since the traditional attitude that "listening to a non-humansinging voice is worthless" was broken down, it is just a matter oftime before technologies break down the prejudice "listening to a
non-human composition is worthless."
When we enjoy a song whose main vocal is generated by asinging synthesis technology on existing video-sharing services like
Niconico (http://www.nicovideo.jp/video_top/) andYouTube (http://www.youtube.com).itis already impossible
to tell from the sound whether the creator of the song is a humanbeing or a computer. In the past, listeners could easily detect humanintervention from human vocals even if the sounds of all the musicalinstruments could be synthesized with such high quality that non­
specialists could not tell the difference from human performances.
However, now that synthesized singing voices have come to be ac­
cepted for main vocals, we often enjoy music in an environment inwhich such human intervention is no longer obvious and we cannot
notice even if all the videos posted on a video-sharing service are
created by a completely automatic system. I
Such an environment holds great potentials. For example, it isconceivable that people could listen to a posted work with music(lyrics, singing, performance) automatically generated by computerand make time-synchronized user comments without noticing the au­
tomated aspect of that work. Then, on the computer side, algorithms
could automatically observe the number of plays, analyze the con­tent of those comments, and use that information as a basis for mak­ing improvements. This capability could therefore make it possible
to repost a work any number of times while making improvements
resulting in the generation of even better music.
How would we as human beings respond if computers began
singing in the true sense of the term? That is, how would we respondif a computer were in some sort of state that could be associated withemotion and began to express itself through singing as a form of self­
assertion? Additionally, if we were to be uncomfortable with the

5. ENLARGING THE CONTRIBUTION OF MUSICINFORMATION RESEARCH TO SOCIETY

1 In actuality, though, there are still no complete technologies for totally

The use of music technologies in industry and society will continue
to expand as various component technologies advance, and music
information research will contribute to society as an essential tech-automating even non-singing tasks like composition/performance and mas­tering in a human manner.

So human intervention could also be detectedfrom that evidence.

11

In the future, then, how will people's deep emotion be elicitedwhen further advances in technology make automatic creation pos­sible and make it easy for anyone to create and perfonn music? If
people's emotion is elicited only by the content itself, nothing will
be changed. On the other hand, if Olympics-type deep emotion is in­dispensable, people cannot give a high evaluation to a work known
to have been created so easily. If, however, people could accept acontext in which a certain virtual character (like a virtual idol orcomputer-generated pop star) creates a musical piece, both content­
induced deep emotion and context-driven deep emotion may be trig­
gered.
A similar discussion has arisen with our singing synthesis tech­
nology called VocaListener [43], which can easily synthesize a natu­
ral singing voice by having the user simply sing (singing-to-singing
synthesis). Given a song produced by such natural singing synthesis,
it is interesting to note that there are people who would give the songa high evaluation solely on the basis of the content itself as well as
people who would give the song a poor evaluation simply because
they knew the song was created by the power of technologi . There
may also be cases in which such a song using VocaListener could
be highly evaluated by people who feel context-driven deep emo­tion after learning of our many years of research activities trying tocontribute to music culture and industry.

nology. In the following, I would like to discuss the questions "Whatis music-induced deep emotion?" and "How can music information
research contribute to human happiness and value enhancement?"
as two fundamental human problems. I point out that this discussion
will be centered on hypotheticals, but my aim here is to dissemi­nate ideas as a foundation for enabling music infonnation research
to make even bigger contributions to society.
5.1. What is music-induced deep emotion?

What kind of deep emotion could be induced by music created by
the automatic-creation technologies described in Section 4.1? How
would music created through the content-creation support technolo­
gies as described in Section 3 be accepted? If such music were
simply wonderful, could automatically created music touch people's
deep emotion? Or even if the quality of such music were high, would
listeners be incapable of having emotionally touching experiences
once they found out that the music was a non-human creation?
In this regard, I hypothesize that at least three kinds of deep
emotion relevant here.
(i) Content-induced deep emotion
This is deep emotion that is elicited by the content itself: thatis, emotion induced without relation to the process used tocreate that content or to any social or cultural significancethat the content may have.

5.2. How can music information research contribute to humanhappiness and value enhancement?

(ii) "Olympics-type" deep emotion
This is deep emotion that is elicited because a fellow humanbeing is the creator of the content in question. For example,
given a piece of music that includes incredibly agile play­ing of some instrument, no emotion at all may be felt if itwere known that the piece is automatically performed by acomputer, but emotion could arise if a human being were per­forming the same musical piece in front of the audience. Thisis analogous to feeling no emotion if a human being and an
automobile were to compete in an Olympic event and the au­
tomobile reached the finish line first.

One of the goals that the field of music infonnation research should
be aiming for over the long-term is to contribute to human happinessthrough the power of music technologies. In this regard, I hypothe­
size that "happiness" is determined by the derivative (slope) of a per­son's psychological perception of reality and not by that person's ab­
solute amount of resources. This hypothesis originates from obser­
vations and introspection and has its limits, but even someone having
only a small amount of resources can feel happy if the derivative of
his or her psychological perception of reality is positive. Conversely,
even someone having a large amount of resources cannot feel happy
if the derivative of his or her psychological perception of reality is
negative. "Sustainable development" is important to a society since
it is then easy to make the value of this derivative positive; if thesociety is only "sustainable," the derivative is zero.

(iii) Context-driven deep emotion
This is deep emotion that is elicited when one knows the con­text in which the content came about. Deep emotion might
be induced when one knows the social, cultural, or personal
background in which the content was created. When the con­tent was created in an unusual situation, for an unusual rea­son, or by a person overcoming a difficulty, it could elicit deep
emotion. Even the "Olympics-type" deep emotion describedabove could be interpreted as context-driven deep emotion.

5.2.1. EnhanCing value per unit resource

I believe that our new concept of "happiness per unit energy" (hap­piness -;- energy) will become important in the future. Media con­tent like music is indispensible to leading an emotionally enriching
life; having a means of obtaining happiness and a sense of fulfill­
ment is directly connected to the quality of life. In particular, music
content is high-quality entertainment requiring a minimal amount ofenergy while being resistant to repeated listening - the amount of
resources and energy used in the production of music have dropped
with the proliferation of digital content production environments. In
this regard, N-th generation of derivative creation can be viewed as
an effective means of condensing the good points of multiple itemsof existing content and increasing happiness. There is also an aspectof recycling (reusing) content here, which means that 2nd genera­tion (secondary or derivative) content can be interpreted as a good

Although here I take up only the above three factors, there areother factors that could contribute to eliciting deep emotion. For
example, deep emotion may be elicited by an experience shared withanother person when enjoying the content together. While this issimilar to context-driven deep emotion, the context in this case is on
the side of the audience rather than that of the content.
In the past, it was difficult to divide up deep emotion into theabove factors (i)--(iii). Whenever content was presented, it was easy
to see who was presenting that content and in what form. Nowadays,
however, given the common practice of posting self-created content
on a video-sharing service, an anonymous work has a chance to behighly evaluated if it induces deep emotion that depends solely on
the content itself. On the other hand, if the content creation process
is disclosed, the possibility of Olympics-type deep emotion could in­crease. lithe anonymous work is presented along with some context,
context-driven deep emotion might be generated.

2 When VocaListener is used, however, the creator has much leeway inadjusting and manipulating musical expression. The more people know about
VocaListener, the more probable it is that Olympics-type deep emotion willbe triggered.

12Such a step-by-step process of improvement has the possibility of
inducing in creators a long-term feeling that the derivative of their
psychological perception of reality is positive.
I do not claim that all of the ideas I have presented above arenew, as I can imagine that there are others who have made similar
hypotheses. That being said, the importance of music information
research to the happiness of mankind is unchanged.

and energy-efficient means of producing content as an unintended
consequence. Research-and-development efforts devoted to increas­ing "happiness per unit energy" (i.e., reducing energy consumption
while increasing happiness) should become increasingly importantin the years to come.
This "happiness per unit energy" can be extended to "value perunit resource" as a more general concept. In the past, only time andcost (including those of human resources) were usually considered
as unit resources and value enhancement was focused on improving
productivity. In today's world, however, it is important that we con­sider energy in addition to time and cost while trying to increase ourhappiness.

6. CONCLUSION

With the hope that music information research will advance to evenhigher levels and open up new possibilities in the future, I haveventured to present some stimulating discussions in this paper. Al­
though I have focused on singing synthesis, Augmented Music­
Understanding Interfaces, and the OngaCREST Project as examples
to advance my arguments, there are many other research topics thatcan be discussed in the same context. I expect music information re­search based on signal processing to be expanded dramatically by thediverse contributions of many researchers. In particular, cooperationand in-depth discussions among researchers of various communities
are essential for human happiness and for value enhancement. I hope
to see further developments in music information research through amelding of diverse fields such as signal processing, machine learn­ing, web service, human-computer interaction, speech processing,hearing, psychoacoustics, and image processing.

5.2.2. Facilitating contributions by music information research

In the information society, one's psychological perception of reality
can be greatly affected not just by physical space but also by infor­
mation space. There is therefore much room for music technology(or information technology on the whole) to make a contribution. It
is easier in information space than in physical space to create condi­
tions that can avoid resource competition or conflict, and as a result,technology that aims to enhance value here holds great possibilities.
Optimizing or maximizing value, however, is not necessarily our true
goal here. If we were to actually succeed in maximizing value, value
could then only decrease and the derivative of one's psychologicalperception of reality would be negative. Furthermore, attempting tomaximize value often requires a great amount of resources.
Consequently, if the above ideas are correct, answering the ques­tion "How can people be made to continually feel over the long termthat the derivative of their psychological perception of reality is pos­
itive?" becomes a challenge that must be faced. Although a psy­chological perception of reality corresponds to no physical quantity,
an illusion of such may be created and effective. If we take thisperspective when supporting people by using music technology, anapproach different from that in the past may appear. For example, in
the "Training" described in Section 3, a positive derivative can be felt
over the long term because ability increases gradually. Even if there
were a technology that could instantly enable a person the best in theworld at some ability, it would not make that person happy because
after that the derivative would no longer be positive. In the "Support
with an immediate effect" approach described in Section 3, the tech­
nology will enhance ability only temporarily, but its temporarinesswould be beneficial since a positive derivative in feeling an improve­
ment in ability can be obtained again and again as many times as theuser quits using and then reuses that technology (however, we cansee the importance of making it difficult to sense that the derivative
has become negative at the instant that the user quits).
Additionally, given the ongoing, monotonic increase in the vol­ume of content, we have to consider methods for achieving a statein which the derivative of the psychological perception of reality
is positive for new content creators. For example, venues for pre­senting content currently take the form of nationwide or worldwide
conventions, but it could be beneficial here to adopt the approachtaken in the sports world, where both national and local contests areheld. This is because a new creator at a national convention may
have to compete with many other people of different levels of ability
in terms of content quality, which raises the possibility that the cre­ator's content will be buried (resource competition easily occurs). A
local convention, in contrast, is apt to feature many creators of sim­
ilar ability, which can foster an environment of friendly competitionin which a creator's content can be appreciated without being buried(resource competition hardly occurs). Popularization of such local
conventions should therefore have a positive effect for new creators.

Acknowledgments: I thank (in alphabetical order by surname) Hi­
romasa Fujihara, Katsutoshi Itoyama, Tomoyasu Nakano, Hiroshi G.
Okuno, and Kazuyoshi Yoshii, who have worked with me to build
the systems presented in this paper. This work was supported in partby CREST, JST.

7. REFERENCES[1] M. Goto, T. Saitou, T. Nakano and H. Fujihara, "Singing infor­
mation processing based on singing voice modeling," in Proc.
ofiCASSP 2010, pp. 5506-5509, 2010.
[2] M. Goto, "Singing information processing," in Proc. of the
12th IEEE International Conference on Signal Processing(IEEE ICSP 2014) , 2014.
[3] Crypton Future Media, "What is the HATSUNE MIKU move­ment? "
http : //www crypton co jp/download/
pdf/info miku e pdf 2008.
[4] H. Kenmochi, "VOCALOID and Hatsune Miku phenomenon
in japan," in Proc. of the First Interdisciplinary Workshop on
Singing Voice (InterSinging 2010), pp. 1-4, 2010.
[5] H. Kenmochi and H. Ohshita, "Vocaloid - commercial
singing synthesizer based on sample concatenation," in Proc.
of Interspeech 2007, pp. 4009-4010, 2007.
[6] M. Goto, "Grand challenges in music information research,"
in Dagstuhl Follow-Ups: Multimodal Music Processing(M. Muller, M. Goto and M. Schedl, eds.), pp. 217-225,
Dagstuhl Publishing, 2012.
[7] S. Fukayama, D. Saito and S. Sagayama, "Assistance fornovice users on creating songs for Japanese lyrics," in Proc.
.

_

_

.

.

.

,

ofICMC 2012,2012.

[8] D. M. Howard and G. F. Welch, "Microcomputer-based
singing ability assessment and development," Applied Acous­
tics, vol. 27, pp. 89-102, 1989.
[9] T. Nakano, M. Goto and Y. Hiraga, "MiruSinger: A singing
skill visualization interface using real-time feedback and mu­
sic CD recordings as referential data," in Proc. of ISM 2007
Workshops (Demonstrations), pp. 75-76, 2007.
13

[10] D. Hoppe, M. Sadakata and P. Desain, "Development of real­time visual feedback assistance in singing training: a review,"
Journal of Computer Assisted Learning, vol. 22, pp. 308-316,
2006.
[11] Y. Takegawa, T. Terada and T. Tsukamoto, "Design and imple­
mentation of a piano practice support system using a real-timefingering recognition technique," in Proc. ofICMC 2011,2011.
[12] C. Oshima, K. Nishimoto and N. Hagita, "A piano duo support
system for parents to lead children to practice musical perfor­
mances," ACM Transactions on Multimedia Computing, Com­
munications, and Applications, vol. 3, no. 2, 2007.
[13] S. Ferguson, A. Vande Moere and D. Cabrera, "Seeing sound:
Real-time sound visualisation in visual feedback loops used fortraining musicians," in Proc. of the 9th International Confer­ence on Information Visualisation, pp. 97-102, 2005.
[14] 1. Yin, Y. Wang and D. Hsu, "Digital violin tutor: An inte­
grated system for beginning violin learners," in Proc. of ACM
Multimedia 2005, pp. 976-985, 2005.
[15] D. Fober, S. Letz and Y. Orlarey, "VEMUS - feedback andgroupware technologies for music instrument learning," inings," in Proc. of SMC 2013, pp. 115-122, 2013.
[28] M. Goto, "Augmented music-understanding interfaces," inProc. of SMC 2009 (Inspirational Session),2009.
[29] M. Goto, "Music listening in the future: Augmented Music­
Understanding Interfaces and Crowd Music Listening," inProc. of the AES 42nd International Con! on Semantic Audio,

pp. 21-30, 2011.
[30] M. Goto, "SmartMusicKIOSK: Music listening station withchorus-search function," in Proc. of UIST 2003, pp. 31-40,
2003.
[31] M. Goto, "A chorus-section detection method for musical au­
dio signals and its application to a music listening station,"
IEEE Trans. on ASLP, vol. 14, no. 5, pp. 1783-1794, 2006.
[32] H. Fujihara, M. Goto, J. Ogata and H. G. Okuno, "LyricSyn­
chronizer: Automatic synchronization system between musi­cal audio signals and lyrics," IEEE Journal of Selected Topics
in Signal Processing, vol. 5, no. 6, pp. 1252-1261, 2011.
[33] K. Itoyama, M. Goto, K. Komatani, T. Ogata and H. G. Okuno,"Instrument equalizer for query-by-example retrieval: Improv­
ing sound source separation based on integrated harmonic andinharmonic models," in Proc. of ISMIR 2008, pp. 133-138,
2008.
[34] K. Yoshii, M. Goto, K. Komatani, T. Ogata and H. G. Okuno,"Drumix: An audio player with functions of realtime drum­
part rearrangement for active music listening," IPSJ Journal,
vol. 48, no. 3, pp. 1229-1239, 2007.
[35] M. Goto, "Active music listening interfaces based on signal
processing," in Proc. of ICASSP 2007,2007.
[36] M. Goto, K. Yoshii, H. Fujihara, M. Mauch and T. Nakano,"SongIe: A web service for active music listening improvedby user contributions," in Proc. of ISMIR 20l l, pp. 311-316,
2011.
[37] M. Goto, 1. Ogata, K. Yoshii, H. Fujihara, M. Mauch and
T. Nakano, "PodCastle and Songle: Crowdsourcing-based web
services for retrieval and browsing of speech and music con­tent," in Proc. of the First International Workshop on Crowd­
sourcing Web Search ( CrowdSearch 2012), pp. 36-41, 2012.
[38] M. Hamasaki and M. Goto, "Songrium: A music browsing as­sistance service based on visualization of massive open collab­
oration within music content creation community," in Proc. ofProc. of SMC 2007,2007.

[16] M. Sadakata, D. Hoppe, A. Brandmeyer, R. Timmers and
P. Desain, "Real-time visual feedback for learning to perform
short rhythms with expressive variations in timing and loud­ness," Journal of New Music Research, vol. 37, no. 3, pp. 207220, 2008.
[17] T. Knight, N. Bouillot and 1. R. Cooperstock, "Visualization
feedback for musical ensemble practice: A case study onphrase articulation and dynamics," in Proc. of Visualization
andData Analysis 2012,2012.

[18] F. Pachet, "The Continuator: Musical interaction with style,"
Journal of New Music Research, vol. 32, no. 3, pp. 333-341,
2003.
[19] F. Pachet and P. Roy, "Markov constraints: steerable generationof Markov sequences," Constraints, vol. 16, no. 2, pp. 148172, 2011.
[20] M. McVicar, S. Fukayama and M. Goto, "AutoRhythmGui­tar: Computer-aided composition for rhythm guitar in the tab
space," in Proc. ofICMC SMC 2014, pp. 293-300, 2014.
[21] M. McVicar, S. Fukayama and M. Goto, "AutoLeadGuitar:
Automatic generation of guitar solo phrases in the tablature
space," in Proc. of the 12th IEEE International Coriference on
Signal Processing (IEEE ICSP 2014), pp. 599-604, 2014.
[22] T. M. Nakra, "Synthesizing expressive music through the lan­
guage of conducting," Journal of New Music Research, vol. 31,
no. I , pp. 11-26, 2002.
[23] 1. Patten, B. Recht and H. Ishii, "Interaction techniques for mu­
sical performance with tabletop tangible interfaces," in Proc. of
ACE 2006,2006.
[24] S. Jordll, G. Geiger, M. Alonso and M. Kaltenbrunner, "Thereactable: Exploring the synergy between live music perfor­
mance and tabletop tangible interfaces," in Proc. of the 1st

the 9th International Symposium on Open Collaboration ( Wik­iSym + OpenSym 2013), pp. 1-10, 2013.
[39] M. Hamasaki, M. Goto and T. Nakano, "Songrium: A musicbrowsing assistance service with interactive visualization andexploration of a web of music," in Proc. of the 23rd Interna­
tional World Wide Web Conference ( W W W 2014), pp. 523-528,
2014.
[40] M. Goto, "Music scene description project: Toward audio­
based real-time music understanding," in Proc. ofISMIR 2003,
pp. 231-232, 2003.
[41] M. Goto, "A real-time music scene description system:
Predominant-FO estimation for detecting melody and bass lines
in real-world audio signals," Speech Communication, vol. 43,
no. 4, pp. 311-329, 2004.
[42] K. Yoshii and M. Goto, "MusicCommentator: Generating
comments synchronized with musical audio signals by a joint
probabilistic model of acoustic and textual features," in Proc.
of ICEC 2009, pp. 85-97, 2009.
[43] T. Nakano and M. Goto, "VocaListener: A singing-to-singing
synthesis system based on iterative parameter estimation," inProc. of SMC 2009, pp. 343-348, 2009.

International Coriference on Tangible and Embedded Interac­tion, pp. 139-146, 2007.
[25] T. Baba, M. Hashida and H. Katayose, "VirtuaIPhilharmony: A
conducting system with heuristics of conducting an orchestra,"in Proc. of NIME 2010, pp. 263-270, 2010.
[26] K. Saino, H. Zen, Y. Nankaku, A. Lee and K. Tokuda, "An
HMM-based singing voice synthesis system," in Proc. of In­
terspeech 2006, pp. 1141-1144, 2006.
[27] T. Nakano and M. Goto, "VocaRefiner: An interactive singing
recording system with integration of multiple singing record-

14