ICSP2014 Proceedings

SINGING INFORMATION PROCESSING

Masataka Goto
National Institute of Advanced Industrial Science and Technology (AIST), Japanm.goto [at] aist.go.jp

ABSTRACTThis paper introduces singing information processing, which is de­fined as music information processing for singing voices. As many
people listen to music with a focus on singing, singing is one of
the most important elements of music. Singing information process­ing attracts attention not only from a scientific point of view but also
from the standpoint of commercial applications,such as singing syn­thesis, automatic singing pitch correction, query-by-humming, andsinging skill evaluation for karaoke. The concept of singing infor­
mation processing is broad and still emerging.
Index Terms- Singing information processing,music informa­tion research

1. INTRODUCTION

As music information research [1-4] has continued to develop, re­search activities related to singing have become more vigorous. Such
activities are attracting attention not only from a scientific pointof view, but also from the standpoint of commercial applications.
Singing-related research is highly diverse, ranging from basic re­search on the features unique to singing to applied research such asthat on singing synthesis,lyrics recognition, lyrics alignment, singer
identification, retrieval of singing voices, singing skill evaluation,
singing training, singing voice conversion, singing impression esti­
mation, and singer robot. I have named this broad range of singing­
related studies singing information processing [5] .
Since singing is one of the most important elements of music,
singing information processing has a major impact on society fromthe viewpoints of industry and culture. In fact, automatic pitch­correction technology for vocals is already being used on a rou­tine basis in the production of commercial music (popular music,
in particular). It has become essential for correcting pitch at points
in a song where the singer is less than skillful and for achieving adesired artificial effect. A function for evaluating (scoring) a per­son's singing in the karaoke industry is also popular. More recently,
singing-synthesis systems have become widely used and people ac­
tively enjoy songs with synthesized singing voices as the main vo­cals.
In this paper, I treat all music-related sounds uttered from a per­son's mouth - whether they are generated by regular singing or evenby "vocal percussion " (mimicking drum sounds) - as "singing". In
the following, I describe this research field of singing information
processing by introducing several systems we have developed.
2. SINGING SYNTHESIS

Achieving the synthesis of singing voices is one of the oldest re­search problems [6] taken up in the field of singing information pro-

978-1-4799-2186-7114/$31.00 ©2014 IEEE

2431

cessing. In recent years, the research into singing synthesis has de­veloped in diverse ways.
Singing synthesis has been actively studied by European re­search institutions, especially in the second half of the 1980s andthroughout the 1990s. Much of this research has focused on syn­thesizing ever better opera singing and to clarify the mechanism ofopera singing through such attempts at synthesis. Representative
examples of this research include synthesized singing voices of avirtual castrated male singer (castrato) created by IRCAM (Institutefor Research and Coordination AcousticlMusic) in France for themovie "Farinelli II Castrato " [7], a singing-synthesis system using
formant synthesis by Sundberg at KTH (Sweden's Royal Instituteof Technology) [8] , and a singing-synthesis system using a physi­
cal model of an articulatory vocal tract by Cook [9] . More recently,
singing-synthesis systems based on corpus-based synthesis methods
have been proposed [10-16] . These methods include concatena­tive synthesis methods and hidden Markov model (HMM) synthesis
methods. Other examples include a system for analyzing, process­ing,and synthesizing scat singing [17], singing morphing that blendsdifferent singing styles [18-20], and singing synthesis that imitates
the user's singing performance [21,22].
Since 2007, singing synthesis technology represented by
Yamaha's VOCAL Ol D [12] has gained much attention in Japan.
Both amateur and professional musicians have started to use singing
synthesizers as their main vocals, and songs sung by computer
singers rather than human singers have become popular and are
now being posted in large numbers on video-sharing services like
Niconico (http://www.nicovideo.jp/video_top/) andYouTube (http://www.youtube.com) [23,24]. Compact discs
featuring compilations of songs created using singing-synthesis
technology are often sold and appear on popular music charts inJapan [25]. In particular, Hatsune Miku [26] is the most popular
software package based on VOCALOID and features a cute synthe­
sized voice with an illustration of a cartoon girl. Although Hatsune
Miku is a virtual singer, she has already performed in many 'live'
concerts with human musicians.
When synthesizing singing voices, most research approaches
have focused on text-to-singing (lyrics-to-singing) synthesis [10,12,13] . In the following, I introduce two other approaches, speech-to­
singing synthesis [27] and singing-to-singing synthesis [22].
2.1. SingBySpeaking: Speech-to-singing synthesis
SingBySpeaking is a speech-to-singing synthesis system that can
synthesize a converted singing voice when given a speaking voice
reading the lyrics of a song and the song's musical score [27] .
This system is based on the STRAIGHT speech manipulation
system [28] and comprises three models controlling three acoustic
features unique to singing voices: the fundamental frequency (FO),phoneme duration, and spectrum. Given the musical score and its

HumanSingerLyricsSinging voiceƗƄƆƋƌƇƒƐƄƕƘőőő

Singing synthesizer1

Singing analysis

Singing synthesizer2

Singer DB (A)

3. LYRIC TRANSCRIPTION AND SYNCHRONIZATION

Singer DB (B)VocaListenerƗƄŃŃŃƇƒŃŃƕƘŃŃƎƌ

Lyricsalignment ƆƋƌŃƐƄŃƗƒ

ƐƄŃŃŃƉƘŃŃŃŃƉƘŃŃŃŃŃƎƄŃƈŃŃƕƘ
ƗƄŃŃŃƗƒŃŃŃŃŃƕƌ

F0
power
time

Parameter estimation
Iterative estimation
until the synthesized singing
becomes closer to singing
by a human singer

time

time

Singing synthesis

Fig. 1. VocaListener: Singing-to-singing synthesis [22,30] .

tempo, the FO control model generates the natural FO contour. The
duration control model lengthens the duration of each phoneme byconsidering the duration of its musical note. The spectral control
model controls both the singing formant [29] and the amplitude mod­ulation of formants in synchronization with vibrato [27].

2.2. VocaListener: Singing-to-singing synthesis
VocaListener (Figure 1) is a singing synthesis system that imitates
the pitch and dynamics of a target human singing [22, 30] . We
call this approach singing-to-singing synthesis. With the help of
the given lyrics of the song being sung, it automatically estimatesthe musical score of the song from the target singing. Because Vo­
caListener can estimate expressive control parameters of a commer­
cial singing synthesizer based on Yamaha's VOCALOID technol­ogy [12] , it easily synthesizes various singing voices that have iden­
tical pitch, dynamics, and lyrics, but different timbres. Thanks tothe estimated natural expressions of the target human singing, syn­thesized singing voices can be human-like and natural without time­
consuming manual adjustment of the control parameters. The sys­
tem also has functions to help modify the user's singing by correct­
ing off-pitch phrases or changing vibrato. Temporal timbre changesof the target singing, however, are not imitated.

As an extension of VocaListener, we then developed VocaLis­tener2 [31] ,a system that imitates timbre changes, not only the pitchand dynamics, of the target human singing. Given a song, VocaL­
istener2 constructs a voice timbre space by using various singing
voices that are synthesized by VocaListener to have the identical
pitch, dynamics, and lyrics, but different timbres. Temporal timbrechanges of the target singing are represented as a trajectory in thisvoice timbre space, and the trajectory is used to synthesize imitatedsinging voices.
Furthermore,for a robot singer, we developed VocaWatcher [30,32] , a system that imitates facial expressions of a human singer's
face during singing by analyzing a video clip of a person singing
that is recorded by a single video camera. VocaWatcher can control
the mouth, eye, and neck motions of a biped humanoid robot, theHRP-4C [33], by imitating corresponding human motions that are
estimated without using any markers in the video clip. The HRP-4C
has a realistic female facial appearance and body shape (160 heightand 46 kg weight with 44 degrees of freedom). The imitated facial
motions can be precisely synchronized, at a phoneme level, withsynthesized singing voices by using the phoneme timing provided
by VocaListener.

Considering that lyrics make up one of the most important elements
of information conveyed by singing, there are various research ap­
proaches that aim to give computers the ability to understand lyrics
in different ways.
Transcribing lyrics from musical audio signals - that is, auto­
matically writing out the lyrics of an unknown song from the audio
signals of that song - is a challenging research problem. This prob­
lem can be treated as the singing version of automatic speech recog­
nition. When targeting singing, however, difficulties such as large
fluctuations and musical accompaniment come into play. Because of
these difficulties, high-performance lyric transcription applicable tosinging that includes accompaniment has yet to be realized. How­ever, since the ability to recognize lyrics could make a variety ofcompelling applications possible such as song retrieval by lyrics,re­search in this area is expected to accelerate in the years to come.
Examples of research targeting singing without musical ac­
companiment include an automatic music-transcription system withlyrics recognition for a solo singing voice [34] , a robust speech­modeling method applicable to even high-pitch signals such as thosein singing [35], query-by-humming music retrieval using not only
pitch (melody) but also lyrics [36], and a method that exploits thefact that the lyrics of popular songs are often repeated in a cho­rus [37]. However,research on singing that includes musical accom­
paniment is rare,though research has been done on the recognition ofphonemes in singing under the limited conditions of known phoneme
boundaries [38]. Additionally, while not strictly lyrics recognition,
research has been reported on indentifying the sung language [39].
If the text of lyrics is known in advance, we can take the lyric
synchronization approach (i. e. , Iyrics-to-audio alignment approach).
This means assigning a temporal correspondence between lyrics andmusical audio signals of a song. For example, we can use an HMM­
based forced-alignment method as is applied in speech recognition,
but this suffers from difficulties unique to singing. When compared
to lyric transcription, lyric synchronization presents a simpler prob­
lem so it can be achieved at a practical level of performance withrelative ease and, except for some cases [40], it can target singing
with musical accompaniment. Research into lyric synchronization
targeting singing with accompaniment can be divided into two cate­gories: that using no forced alignment [41-43] and that using forcedalignment [44-48] . When textual chord information is additionallygiven, it can be used to improve the accuracy of lyric synchroniza­tion [49].
In the following, I introduce three lyric-based systems, L yric­Synchronizer, Hyperlinking L yrics, and VocaRefiner.
3.1. LyricSynchronizer:

Automatic synchronization of lyrics

with polyphonic music recordings

L yricSynchronizer (Figure 2) is a system that displays scrolling
lyrics with the phrase currently being sung highlighted during play­back of a song [47,48] . Because the lyrics are automatically syn­chronized with the song, a user can easily follow the current play­back position even on a small screen. Moreover, a user can click on
a word in the lyrics shown on a screen to jump to and listen from thatword.
Achieving this is difficult because most singing voices are ac­companied by other musical instruments. It is therefore necessary
to focus on the vocal part in polyphonic sound mixtures by reducing
the influence of accompaniment sounds. To do this, the system first
segregates the vocal melody from polyphonic sound mixtures, de-2432

LyricSynch ronizer:
utomatic::s),ncbrollization between music and lyricse I n e r by Tomoyasu NAKANO and Masataka GOTO IAIST)by Hiromasa Fujihara. Masataka Goto and Hiroshi G. Okuno

Angel Baby / Betty
RWC Music Database: Popular Music
-{}

01:26

Come BackTo Piece Of My Place I Said You
ToTakeThis Love So Plain And True Uh
Angel Baby You're Like An Angel Baby
Angel Baby I Want You Angel Baby
Angel Baby You're Like An Angel Baby
Angel Baby I Want You Angel Baby
Won't You Be My Girl

Proje<t

Vocal

Edit

c!J6 6 (ilEl •• �

Need You
Love Is A Fantasy Lovin You

Fig. 3.

Thinking Of You Everytime Everywhere

VocaRefiner: Interactive singing recording by integrating
mUltiple singing recordings [51].

Fig. 2. LyricSynchronizer: Automatic synchronization of lyrics with

polyphonic music recordings [47,48] .
4. VOCAL TIMBRE ANALYSIS

tects vocal sections, and then applies the Viterbi alignment (forcedalignment) method to those sections to locate each phoneme [47,48] .

3.2. Hyperlinking Lyrics: Creating hyperlinks between phrases
in song lyrics

Hyperlinking L yrics is a system for creating a hyperlink from aphrase in the lyrics of a song to the same phrase in the lyrics ofanother song [50]. This can be used in various applications, such as
song clustering based on the meaning of the lyrics and a music play­back interface that will enable a user to browse and discover songson the basis of lyrics.
Given a song database consisting of songs with their text lyrics
and songs without their text lyrics, the system first extracts appro­
priate keywords (phrases) from the text lyrics without using audio
signals. It then estimates the start and end times of these keywords
in audio signals by using HMMs.

3.3. VocaRefiner: Interactive singing recording by integrating
multiple singing recordings

VocaRefiner (Figure 3) is a system for enabling a singer to make abetter singing recording by integrating multiple recordings of a songhe or she has sung repeatedly [51]. It features a function called click­able lyrics, with which the singer can click a word in the displayed
lyrics to start recording from that word as can be done on LyricSyn­
chronizer. Clickable lyrics facilitate efficient mUltiple recordings be­cause the singer can easily and quickly repeat recordings of a phrase
until satisfied. Each of the recordings is automatically aligned tothe music-synchronized lyrics for comparison by using a phoneticalignment technique.

VocaRefiner also features a function, called three-element de­composition, that analyzes each recording to decompose it into threeessential elements: FO, power, and spectral envelope. This enables
the singer to select good elements from different recordings and use
them to synthesize a better recording by taking full advantage of thesinger's ability. Pitch correction and time stretching are also sup­ported.

The timbre of a singing voice is an important element in singing not
only because it can provide information as to who is actually singing
but also because it can greatly affect the impression a person receiveson listening to a singing voice [52].
Many studies on identifying the singer of a song have been madesince the beginning of the 2000s [53--64] . These studies aim to spec­
ity the singer name for an input song from a set of singers regis­
tered beforehand in a database. This is generally accomplished byfirst extracting features and then applying a machine-learning tech­nique. Some studies apply a vocal-activity detection method anduse only the segments judged to have singing for identification pur­poses [54-56,58,61], other studies reduce the effects of musical ac­
companiment in addition to vocal-segment detection [57,59,63,64],and still another study makes use of the individual characteristics ofvibrato [60].
Methods that can estimate vocal-timbre similarity between
singing voices in different songs - i. e., determine the extent towhich a singing voice in one song resembles that in another song- have a wide range of applications. A variety of methods havebeen proposed, including one on the automatic clustering of songs
based on similarity between singing voices [65] and another on vo­
cal timbre analysis based on topic modeling using latent Dirichlet
allocation (LDA), which can be used for cross-gender vocal-timbre
similarity [66].
Furthermore, singing voice conversion that changes the timbre
of a singing voice is also a popular research topic and has been stud­ied [67-71] .
In the following,I introduce two systems related to vocal timbre,
Singer JD and VocalFinder.
4.1. Singer ID: Singer identification for polyphonic music
recordings

The Singer ID system automatically identifies the name of the singer
who sang the input song in the form of polyphonic musical audio sig­
nals [63,64] . Even if the singer names for some songs are not avail­able as metadata, the system enables users to retrieve those songs
based on the singer names, for example. This is especially usefulwhen artist names in the metadata are not singer names.
Like LyricSynchronizer, this system also segregates the vocal
melody from polyphonic sound mixtures, and then selects frames2433

tIlocalFinder: MIR system based on vocal timbre similarity

by Hiromasa Fujihara and Masataka Goto

°

Query
T itle: PROLOGUE
Artist: Tomomi OgataoLibrary

Find

Once in a life time
Someday
Don't Say Good bye
No Regrets
Once and For All
Tomomi Ogata
True Heart.

..

tell meHaru ga kurukara
Flyaway
Together
Sagashimono

��"jng Round and RoundYuuichi Nagay
ama
f�,

.

00:58

x

• °

110

•

liD••011.0-.uRetrieved Results
1.
2.

3.

4.

S.
6.

Tomomi Ogata / True Heart..u Tomomi Ogata / Haru ga kurukara
IIII] Eves I Koina Ver.2.4

� Brakes / Getting Over.::6.JJ Rin / Feeling In MyHeart.:II] Hirami Yoshii / Anata to aete
.7.. . . --0-8. � Hirami Yoshii I Magic in your eyes
9. II3BJ Tomomi Ogata / Flyaway
10. � Betty/ Weekend
11. .-:n:J fevers I Jinsei konnamono
12. � Tomomi Ogata / tell me
13. ..::m Rin / Mrs. Maril
14. .:3J Rin / Ashita wa
15 . .r.44l Rin / Harukana ornoi

Expl"ossion Select
Bass
.nom

d o N ...

�

Fig. 4. VocalFinder: Music information retrieval based on singing

voice timbre [64,72] .

that are reliable enough for classification to improve the robustness.
After training of a Gaussian mixture model (GMM) for each singer,
the identity of the singer is determined on the basis of likelihood.

4.2. VocalFinder: Music information retrieval based on singing
voice timbre

VocalFinder (Figure 4) is a music information retrieval system thatcan search a database for songs that have similar vocal timbres [64,72] . Given a query song presented by a user, a list of songs withvocals having similar voice timbre to the query song is shown. With
this system, we can find a song by using its musical content (i.e.,
vocal timbre) in addition to traditional bibliographic information.

To achieve this, we developed a method for extracting feature
vectors that represent the characteristics of singing voices and cal­culating the vocal-timbre similarity between two songs by using themutual information content of their feature vectors.

5. MUSIC INFORMATION RETRIEVAL BASED ONSINGING VOICESGiven the huge number of songs now available, music information
retrieval is becoming increasingly important. The VocalFinder sys­
tem described in Section 4. 2 is an example of music information re­trieval based on singing voices.
"Humming " can be treated as singing without lyrics, and mu­
sic retrieval based on humming, or Query-By-Humming (QBH), isa method of retrieving songs using humming as a search key. The
dawn of research on "content-based music information retrieval " us­ing other than bibliographic information (such as titles and artist
names) occurred in the 1990s [73-75], and it was this research that
set the foundation for many studies on QBH (see [76--79] for details).
A variety of QBH methods have been proposed, including a method
for expressing the dynamic fluctuations of the FO locus by a phaseplane (FO - �FO plane) [80], a method for expressing FO by prob­
abilistic models without estimating FO from the singing voice [81],and a method for improving retrieval performance by learning fromuser singing [82]. There have also been attempts at using featuresother than FO for music retrieval, such as a method that uses the

Tempo

Snare
.nom

�

�

taN"

SelectI Practice I

Mode SelectMusic SelectNo.013��Drums

St artNotation
ArrangeFig. 5. Voice Drummer: Music notation of drums using vocal per­cussion input [89].

lyrics uttered by a singing voice [36] and a method that uses mel­frequency cepstral coefficients (MFCC) or formant frequencies [83].
Another style of music information retrieval is drum-pattern re­trieval by "voice percussion " [84-89] , which refers to a type of
singing that expresses the sounds of a drum by using onomatopoeicwords like "don-tan-do-do-tan " or that acoustically imitates thosesounds in a more faithful manner. The latter is often called "beat­boxing," and Query-By-Beatboxing is a method for recognizing thesounds so uttered and retrieving corresponding drum sounds or adrum pattern. In the following, I introduce a system for this sort of
retrieval, Voice D rummer.
5.1. Voice Drummer: Music notation of drums using vocal per­cussion input

Voice D rummer (Figure 5) is a percussion instrument notation sys­
tem that uses oral percussion patterns as input [89]. A user singsout a drum pattern (beatboxing), which is analyzed and matchedwith entries in a drum pattern database; this analysis and matchingis based on onset timing patterns and intended drum types (bass or
snare drums). As real-time feedback,the system shows the graphicalscore of recognized (retrieved) patterns. The user can also sing along
to an existing musical piece without drums so that its drum patterns
can be arranged according to the sung patterns.
The system uses onomatopoeia as internal representation of
drum sounds, and retrieves a sung drum pattern from the patterndatabase by using HMMs. A pronunciation dictionary of ono­
matopoeic expressions of bass and snare drums is obtained from ex­pression experiments.
6. SINGING SKILL EVALUATION

Singing skill plays an important role in characterizing singing, andclarification of the acoustic features that determine singing skill is
expected to not only benefit singing instruction and training but toalso find application in music information retrieval and singing syn­thesis.
Studies to reveal acoustic features related to singing skill havebeen done. Vibrato and the singer's formant are typical requirements
for a good singing voice [90]. By investigating acoustic and percep­
tual effects of vocal training in amateur singing,it was found that two2434kinds of FO fluctuations (vibrato and overshoot) and singer's formant
were changed by vocal training [91]. Conditions for "vocal ugliness "(unpleasant singing) have also been identified. These include irreg­
ular vibrato, pitch dissonance, weak fundamental amplitude in thevoice source spectrum, and a lack of singer's formant [8] . The man­ner in which breaths are taken while singing and the positioning ofbreath sounds may also be associated with singing skill. Detailed
studies of such an association have not yet been performed, but the
acoustic analysis of breath sounds and their automatic detection havebeen studied [92,93] .
The most common approach for automatically evaluating
singing skill is the karaoke scoring function. This popular function
generally scores a singing voice based mainly on the difference be­
tween that voice and the musical score (pitch) of the song in ques­tion. Moreover, functions for detecting vibrato and portamento havebeen added. Research on automatic evaluation based on agreementof pitch and note duration has also been done [94]. In addition,stud­ies on automatically evaluating singing skills without the use of mu­
sical scores have been made. There is an SV M-based method us­
ing acoustic features related to relative pitch and vibrato [95] and a
GMM-based method using MFCC or mel-filter-bank coefficients as
acoustic features [96]. Singing enthusiasm can also be evaluated bydetecting related acoustic features [97].
In addition to the above automatic scoring function for karaoke,
other applications related to singing skill are being considered, suchas those for improving singing skill (in support of singing training).
In particular,methods have been proposed for visualizing the resultsof user-singer analysis and providing feedback in real time [98-100] .
It has been reported that pitch in singing can be improved by display­
ing the extent to which pitch in the user's singing departs from thetarget pitch [98] .
In the following, I introduce two systems related to the above
topics,MiruSinger and Breath D etection.
6.1. MiruSinger: Singing skill visualization and training
MiruSinger (Figure 6) is a singing skill visualization system that an­alyzes and visualizes vocal singing with reference to the vocal part ofa target song that a user wants to sing more skillfully [100]. As real­time feedback, the system visualizes the characteristics of singing
skills, such as FO (the fundamental frequency) and vibrato sectionsof the user's singing voice, showing comparison with the FO trajec­
tory of the vocal part estimated in polyphonic sound mixtures.
Each vibrato is detected on the basis of our method developed
for automatic singing skill evaluation for unknown melodies, inwhich a sung phrase can be categorized into good or poor classes
by using an SV M [95]. The vocal melody of the target song is also
estimated in polyphonic sound mixtures.

MJRu S, t NGfR by Tomoyasu NAKANO, Masataka GOTO, Yuzuru HIRAGA-•

Referential data}
�;I"� �2i �·-1�".l""=F;::���C: ��/vJi·�; p.. ..... .... . .. ... . )-....-..-.
.. .. -....�..
.,.r�-...

The estimated FO as the vocal_of the music CD recordingroo=: ................. ::r:::"'""'� .--",�__..,..J"�_.-..r-.---, 1 octave difference

.

=
=

.
The FO sung by the user==-"" ....,,1=

Support for judging the correct
FO of the referential data

'�JI��,.,.

�lp037mc den � ::::;0) ,----- �,----VISolo)Analyze I
V(AII)
� � �
� �
Sing

Rererentlal Datarf;;

Listenr solo r All

Draw

J--

Mode

�
Listen I
�_ _ Draw J

Fig. 6. MiruSinger: Singing skill visualization and training [100].

notable spectral peak at about 1. 6 kHz for male singers and 1. 7 kHz
for female singers [93].
7. CONCLUSION

In this paper, I have provided an overview of the research field re­garding singing information processing and have described exam­
ples of our singing information processing systems. It is importantto realize, though, that these examples are just brief excerpts takenfrom a wide variety of systems and methods that have been devel­
oped by many researchers. We expect research related to singing
information processing to progress rapidly in the years to come be­cause every system and method available today still needs further
refinement. A wide variety of research problems not discussed inthis paper remain to be solved. It will be increasingly important thatall kinds of knowledge concerning singing voices - such as psy­chology [101], physiology [102], and vocal pedagogy [103] - beconsidered in combination with signal processing, machine learn­ing, human-computer interaction, and other key technologies. As
this field of research continues to develop in the future, I expect tosee an increasing number of researchers contributing to the advances
realized.
I thank (in alphabetical order by surname)
Hiromasa Fujihara Yuzuru Hiraga Tomoyasu Nakano, Hiroshi G.
Okuno, and Takeshi Saitou, who have worked with me to build thesystems presented in this paper. This work was supported in part by
CREST,JST.

Acknowledgments:

6.2. Breath Detection: Automatic detection of breath sounds inunaccompanied singing voice

Our automatic breath detection system finds each breath sound inunaccompanied solo singing [93] . Detected breath sounds can besuppressed as noise, or can be used as valuable cues for applications
such as the segmentation and structural analysis of music and theevaluation of a singer's skill.
The system uses HMMs with MFCC, �MFCC, and �poweras acoustic features to detect breath sounds as variant time-length
events. We also did a detailed acoustic analysis of breath sounds andfound that the spectral envelopes of breath sounds remain similar
within the same song, and their long-term average spectra have a2435

8. REFERENCES[I] M. Goto and K. Hirata, "Invited review: Recent studies onmusic information processing," Acoustical Science and Tech­
nology (edited by the Acoustical Society of Japan), vol. 25,
no. 6,pp. 419-425, 2004.
[2] A. Klapuri and M. Davy,eds. , Signal Processing Methods forMusic Transcription. Springer, 2006.
[3] M. Goto, "Grand challenges in music information research,"
in D agstuhl Follow-Ups: Multimodal Music Processing(M. Muller, M. Goto and M. Schedl, eds.), pp. 217-225,
Dagstuhl Publishing, 2012.

[4] M. Goto, "Frontiers of music information research based on
signal processing," in Proc. of the 12th IEEE International
Conference on Signal Processing (IEEE ICSP 2014),2014.

[21] J. Janer,J. Bonada and M. Blaauw,"Performance-driven con­trol for sample-based singing voice synthesis," in Proc. of1)AFx-O� pp. 41-44,2006.

[5] M. Goto, T. Saitou, T. Nakano and H. Fujihara, "Singing information processing based on singing voice modeling," inProc. of ICASSP 2010, pp. 5506-5509, 2010.

[22] T. Nakano and M. Goto,"VocaListener: A singing-to-singing
synthesis system based on iterative parameter estimation," inProc. of SMC 2009, pp. 343-348, 2009.

[6] P. R. Cook, "Singing voice synthesis: History, current work,and future directions," Computer Music Journal, vol. 20,
no. 3, pp. 38-46,1996.

[23] M. Hamasaki, H. Takeda and T. Nishimura, "Network anal­
ysis of massively collaborative creation of multimedia con­tents: Case study of Hatsune Miku videos on Nico Nico
Douga," in Proc. of uxTV'08, pp. 165-168,2008.

[7] P. Depalle, G. Garcia and X. Rodet, "A virtual castrato," inProc. of ICMC '94, pp. 357-360, 1994.
[8] J. Sundberg, "The KTH synthesis of singing," Advances in
Cognitive Psychology. Special issue on Music Performance,
vol. 2, no. 2-3, pp. 131-143,2006.
[9] P. R. Cook,Identification of Control Parameters in an Articulatory Vocal Tract Model, With Applications to the Synthesisof Singing. PhD thesis, Stanford University,1991.
[10] J. Bonada and X. Serra, "Synthesis of the singing voice byperformance sampling and spectral models," IEEE Signal
Processing Magazine, vol. 24, no. 2, pp. 67-79, 2007.
[11] D. Schwarz, "Corpus-based concatenative synthesis," IEEE
Signal Processing Magazine, vol. 24,no. 2,pp. 92-104,2007.
[12] H. Kenmochi and H. Ohshita, "Vocaloid - commercial
singing synthesizer based on sample concatenation," in Proc.
of Interspeech 2007, pp. 4009-4010,2007.
[13] K. Saino, H. Zen, Y. Nankaku, A. Lee and K. Tokuda, "An
HMM-based singing voice synthesis system," in Proc. of Interspeech 2006, pp. 1141-1144,2006.
[14] K. Saino, M. Tachibana and H. Kenmochi, "A singing stylemodeling system for singing voice synthesizers," in Proc. ofInterspeech 2010, pp. 2894-2897, 2010.
[15] T. Nose, M. Kanemoto, T. Koriyama and T. Kobayashi, "Astyle control technique for singing voice synthesis based on
multiple-regression HSMM," in Proc. of Interspeech 2013,
pp. 378-382, 2013.
[16] M. Umbert, J. Bonada and M. Blaauw, "Generating singing
voice expression contours based on unit selection," in Proc.
of SMAC 2013,2013.
[17] H. Kawahara and H. Katayose, "Scat singing generation using a versatile speech manipulation system STRAIGHT," The
Journal of the Acoustical Society of America (The 141st Meeting of the Acoustical Society of America), vol. 109, no. 5,
pp. 2425-2426,2001.
[18] H. Kawahara, "Application and extensions of STRAIGHTbased morphing for singing voice manipulations based on
vowel centred approach," in Proc. of the 19th International
Congress on Acoustics 2007 (ICA 2007), pp. 2018-2021,
2007.

[24] Cabinet Office, Government of Japan, "Virtual idol," High­lighting JAPAN through images, vol. 2, no. 11, pp. 24-25,
2009. http://www.gov-online.go.jp/pdf/hlj
img/vol 0020et/24-25.pdf.
[25] H. Kenmochi,"VOCALOID and Hatsune Miku phenomenon
in japan," in Proc. of the First Interdisciplinary Workshop on
Singing Voice (InterSinging 2010),pp. 1-4,2010.
[26] Crypton Future Media, "What is the HATSUNE MIKU
movement?"
http://www.crypton.co.jp/
download/pdf/info_miku_e.pdf, 2008.
[27] T. Saitou, M. Goto, M. Unoki and M. Akagi, "Speech-to­
singing synthesis: Converting speaking voices to singing
voices by controlling acoustic features unique to singing
voices," in Proc. of WASPAA 2007, pp. 215-218, 2007.
[28] H. Kawahara, I. Masuda-Kasuse and A. de Cheveigne, "Re­structuring speech representations using a pitch-adaptive
time-frequency smoothing and an instantaneous-frequency­
based FO extraction: Possible role of a repetitive structure insounds," Speech Communication, vol. 27, no. 3-4, pp. 187207,1999.
[29] J. Sundberg, The Science of Singing Voice. Northern Illinois
University Press, 1987.
[30] M. Goto, T. Nakano, S. Kajita, Y. Matsusaka, S. Nakaoka
and K. Yokoi, "VocaListener and VocaWatcher: Imitating ahuman singer by using signal processing," in Proc. of ICASSP
2012,pp. 5393-5396,2012.
[31] T. Nakano and M. Goto,"VocaListener2: A singing synthesis
system able to mimic a user's singing in terms of voice timbrechanges as well as pitch and dynamics," in Proc. of ICASSP
20ll, pp. 453-456, 2011.
[32] S. Kajita,T. Nakano, M. Goto,Y. Matsusaka, S. Nakaoka andK. Yokoi, "VocaWatcher: Natural singing motion generator
for a humanoid robot," in Proc. of IROS 2011,2011.
[33] K. Kaneko,F. Kanehiro, M. Morisawa,K. Miura, S. Nakaoka
and S. Kajita, "Cybernetic Human HRP-4C," in Proc. ofHu­
manoids 2009, pp. 7-14,2009.
[34] C.-K. Wang, R. -Y. Lyu and Y.-c. Chiang, "An automatic
singing transcription system with multilingual singing lyric
recognizer and robust melody tracker," in Proc. of Eurospeech
2003,pp. 1197-1200,2003.

[19] M. Morise, M. Onishi, H. Kawahara and H. Katayose,"v.morish'09: A morphing-based singing design interface for
vocal melodies," in Proc. of ICEC 2009, pp. 185-190,2009.

[35] A. Sasou, M. Goto, S. Hayamizu and K. Tanaka, "An auto­
regressive,non-stationary excited signal parameter estimation
method and an evaluation of a singing-voice recognition," inProc. ofICASSP 2005, pp. 1-237-240, 2005.

[20] H. Kawahara, M. Morise, H. Banno and V. G. Skuk, "Temporally variable multi-aspect N-way morphing based oninterference-free speech representations," in Proc. of APSIPAASC 2013,2013.

[36] M. Suzuki, T. Hosoya, A. Ito and S. Makino, "Music infor­
mation retrieval from a singing voice using lyrics and melody
information," EURASIP Journal on Advances in Signal Pro­
cessing, vol. 2007, 2007.

2436

[37] M. McVicar, D. P. Ellis and M. Goto, "Leveraging repetition
for improved automatic lyric transcription in popular music,"in Proc. ofiCASSP 2014, pp. 3141-3145, 2014.
[38] M. Gruhne, K. Schmidt and C. Dittmar, "Phoneme recogni­tion in popular music," in Proc. of1SM1R 2007, pp. 369-370,
2007.

[54] A. L. Berenzweig, D. P. W. Ellis and S. Lawrence, "Using
voice segments to improve artist classification of music," inProc. of AES-22 IntI. Con! on Virt. , Synth. , and Ent. Audio,
2002.
[55] Y. E. Kim and B. Whitman, "Singer identificatin in popular
music recordings using voice coding features," in Proc. of IS­
MIR 2002, pp. 164-169,2002.

[39] W.-H. Tsai and H.-M. Wang,"Automatic identification of thesung language in popular music recordings," J New Music
Res. , vol. 36, no. 2, pp. 105-114,2007.

[56] T. Zhang,"Automatic singer identification," in Proc. ofiCME
2003, vol. I, pp. 33-36, 2003.

[40] A. Loscos, P. Cano and 1. Bonada, "Low-delay singing voicealignment to text," in Proc. of ICMC 99, 1999.

[57] M. A. Bartsch, Automatic Singer Identification in PolyphonicMusic. PhD thesis, The University of Michigan, 2004.

[41] C. H. Wong,W. M. Szeto and K. H. Wong,"Automatic lyricsalignment for cantonese popular music," Multimedia Systems,
vol. 4-5, no. 12, pp. 307-323, 2007.

[58] N. C. Maddage, C. Xu and Y. Wang, "Singer identifica­tion based on vocal and instrumental models," in Proc. of
ICPR'04, vol. 2, pp. 375-378, 2004.

[42] M. Mi.iller,F. Kurth, D. Damm, C. Fremerey and M. Clausen,"Lyrics-based audio retrieval and multimodal navigation inmusic collections," in Proc. of ECD L 2007, pp. 112-123,
2007.
[43] M.-Y. Kan, Y. Wang, D. Iskandar, T. L. Nwe and A. Shenoy,"Lyrically: Automatic synchronization of textual lyrics toacoustic music signals," IEEE Transactions on Audio, Speech,
and L anguage Processing, vol. 16, no. 2, pp. 338-349, 2008.
[44] K. Chen, S. Gao,Y. Zhu and Q. Sun, "Popular song and lyrics
synchronization and its application to music information re­trieval," in Proc. ofMMCN'06, 2006.
[45] H. Fujihara, M. Goto, 1. Ogata, K. Komatani, T. Ogata and
H. G. Okuno,"Automatic synchronization between lyrics andmusic CD recordings based on Viterbi alignment of segre­
gated vocal signals," in Proc. of ISM 2006, pp. 257-264,
2006.
[46] D. Iskandar, Y. Wang, M.-Y. Kan and H. Li, "Syllabic levelautomatic synchronization of music signals and text lyrics,"
in Proc. ACM Multimedia 2006, pp. 659-662, 2006.
[47] H. Fujihara and M. Goto, "Three techniques for improving
automatic synchronization between music and lyrics: Frica­
tive detection,filler model,and novel feature vectors for vocal
activity detection," in Proc. of ICASSP 2008, 2008.
[48] H. Fujihara, M. Goto, 1. Ogata and H. G. Okuno, "LyricSyn­
chronizer: Automatic synchronization system between musi­cal audio signals and lyrics," IEEE Journal of Selected Topics
in Signal Processing, vol. 5, no. 6, pp. 1252-1261,2011.
[49] M. Mauch, H. Fujihara and M. Goto, "Integrating additional
chord information into HMM-based lyrics-to-audio align­ment," IEEE Transactions on Audio, Speech, and L anguage
Processing, vol. 20, no. 1, pp. 200-210, 2012.
[50] H. Fujihara, M. Goto and 1. Ogata, "Hyperlinking Lyrics:
A method for creating hyperlinks between phrases in song
lyrics," in Proc. ofiSMIR 2008, pp. 281-286, 2008.
[51] T. Nakano and M. Goto, "VocaRefiner: An interactive singing
recording system with integration of multiple singing record­ings," in Proc. of SMC 2013, pp. 115-122,2013.
[52] A. Kanato, T. Nakano, M. Goto and H. Kikuchi, "An au­
tomatic singing impression estimation method using factor
analysis and mUltiple regression," in Proc. of ICMC SMC
2014, pp. 1244-1251,2014.
[53] B. Whitman, G. Flake and S. Lawrence, "Artist detection inmusic with minnowmatch," in Proc. of NNSP 2001, pp. 559568,2001.
2437[59] w.-H. Tsai and H.-M. Wang, "Automatic singer recognitionof popular music recordings via estimation and modeling ofsolo vocal signals," IEEE Transactions on Audio, Speech, andL anguage Processing, vol. 14, no. 1, pp. 330-341, 2006.
[60] T. L. Nwe and H. Li, "Exploring vibrato-motivated acous­
tic features for singer identification," IEEE Transactions on
Audio, Speech, and L anguage Processing, vol. 15, no. 2,
pp. 519-530,2007.
[61] 1. Shen,B. Cui,J. Shepherd and K.-L. Tan,"Towards efficient
automated singer identification in large music databases," inProc. of SIGIR'06, pp. 59-66,2006.
[62] A. Mesaros, T. Virtanen and A. Klapuri, "Singer identifica­tion in polyphonic music using vocal separation and patternrecognition methods," in Proc. ofiSMIR 2007, 2007.
[63] H. Fujihara, T. Kitahara, M. Goto,K. Komatani, T. Ogata and
H. G. Okuno,"Singer identification based on accompanimentsound reduction and reliable frame selection," in Proc. of IS­
MIR 2005, pp. 329-336, 2005.
[64] H. Fujihara, M. Goto, T. Kitahara and H. G. Okuno, "Amodeling of singing voice robust to accompaniment sounds
and its application to singer identification and vocal-timbre­
similarity-based music information retrieval," IEEE Transac­
tions on Audio, Speech, and L anguage Processing, vol. 18,
no. 3,pp. 638-648,2010.
[65] w.-H. Tsai, H.-M. Wang, D. Rodgers, S.-S. Cheng and H.­
M. Yu, "Blind clustering of popular music recordings basedon singer voice characteristics," in Proc. of ISMIR 2003,
pp. 167-173,2003.
[66] T. Nakano,K. Yoshii and M. Goto,"Vocal timbre analysis us­
ing latent Dirichlet allocation and cross-gender vocal timbre
similarity," in Proc. ofiCASSP 2014, pp. 5239-5343, 2014.
[67] F. Villavicencio and 1. Bonada, "Applying voice conversion
to concatenative singing-voice synthesis," in Proc. of Inter­speech 2010, pp. 2162-2165, 2010.
[68] H. Doi, T. Toda, T. Nakano, M. Goto and S. Nakamura,"Singing voice conversion method based on many-to-many
eigenvoice conversion and training data generation using asinging-to-singing synthesis system," in Proc. of APSIPA ASC
2012, 2012.
[69] H. Doi, T. Toda, T. Nakano, M. Goto and S. Nakamura,"Evaluation of a singing voice conversion method basedon many-to-many eigenvoice conversion," in Proc. of Inter­speech 2013, pp. 1067-1071,2013.

[70] K. Kobayashi, H. Doi, T. Toda, T. Nakano, M. Goto, G. Neu­big, S. Sakti and S. Nakamura, "An investigation of acous­
tic features for singing voice conversion based on perceptual
age," in Froc. of Interspeech 2013, pp. 1057-1061,2013.
[71] K. Kobayashi, T. Toda, T. Nakano, M. Goto, G. Neubig,
S. Sakti and S. Nakamura, "Regression approaches to per­ceptual age control in singing voice conversion," in Proc. of
1CASSP 2014, pp. 7954-7958, 2014.
[72] H. Fujihara and M. Goto, "A music information retrieval sys­
tem based on singing voice timbre," in Froc. of 1SMIR 2007,
pp. 467-470, 2007.
[73] T. Kageyama, K. Mochizuki and Y. Takashima, "Melody re­trieval with humming," in Froc. of ICMC 93, pp. 349-351,
1993.
[74] A. Ghias, 1. Logan, D. Chamberlin and B. Smith, "Queryby humming: Musical information retrieval in an audio
database," in Froc. of ACM Multimedia 1995, vol. 95,
pp. 231-236,1995.
[75] T. Sonoda, M. Goto and Y. Muraoka, "A WWW-based
melody retrieval system," in Proceedings of ICMC 98,
pp. 349-352, 1998.
[76] R. Dannenberg, W. Birmingham, B. Pardo, C. Meek, N. Hu
and G. Tzanetakis, "A comparative evaluation of searchtechniques for query-by-humming using the musart testbed,"
Journal of the American Society for Information Science andTechnology , vol. 58, pp. 687-701,2007.
[77] E. Unal, E. Chew, P. Georgiou and S. Narayanan, "Chal­lenging uncertainty in query by humming systems: A finger­
printing approach," IEEE Transactions on Audio, Speech, andL anguage Processing, vol. 16, pp. 359-371, 2008.
[78] J.-S. R. Jang and H.-R. Lee, "A general frameworkof progressive filtering and its application to query bysinging/humming," IEEE Transactions on Audio, Speech, andL anguage Processing, vol. 16, no. 2, pp. 350--358, 2008.
[79] W.-H. Tsai, Y.-M. Tu and C.-H. Ma, "An FFT-based fast
melody comparison method for query-by-singing/humming
systems," Pattern Recognition L etters, vol. 33, no. 16,
pp. 2285-2291,2012.
[80] Y. Ohishi,M. Goto,K. Ito and K. Takeda,"A stochastic repre­
sentation of the dynamics of sung melody," in Proc. ofiSMIR
2007, pp. 371-372, 2007.
[81] M. Suzuki, T. Ichikawa, A. Ito and S. Makino, "Novel tonal
feature and statistical user modeling for query-by-humming,"
Journal of Information Processing, vol. 17,pp. 95-105,2009.
[82] D. Little, D. Raffensperger and B. Pardo, "A query by hum­ming system that learns from experience," in Proc. of ISMIR
2007, pp. 335-338, 2007.
[83] A. Duda, A. Nurnberger and S. Stober, "Towards query bysinging / humming on audio databases," in Proc. of ISMIR
2007, pp. 331-334, 2007.
[84] O. Gillet and G. Richard, "Drum loops retrieval from spoken
queries," Journal of Intelligent Information Systems, vol. 24,
no. 2-3,pp. 159-177,2005.
[85] O. Gillet and G. Richard, "Indexing and querying drum loops
databases," in Froc. of CBMI 2005, 2005.
[86] A. Kapur, M. Benning and G. Tzanetakis, "Query-by-beat­boxing: Music retrieval for the dj," in Froc. of ISMIR 2004,
pp. 170--177,2004.
2438[87] A. Hazan, "Towards automatic transcription of expressive
oral percussive performances," in Froc. of lUi 2005,pp. 296298,2005.
[88] E. Sinyor, C. McKay, R. Fiebrink, D. McEnnis and I. Fuji­naga, "Beatbox classification using ace," in Proc. of ISMIR
2005,pp. 672-675,2005.
[89] T. Nakano, M. Goto, J. Ogata and Y. Hiraga, "Voice Drum­mer: A music notation interface of drum sounds using voice
percussion input," in Proc. ofUiST 2005 (D emos), pp. 49-50,
2005.
[90] W, T, Bartholomew, "A physical definition of "good voice­quality " in the male voice," J Acoust. Soc. Am. , vol. 55,
pp. 838-844, 1934.
[91] T. Saitou and M. Goto,"Acoustic and perceptual effects of vo­
cal training in amateur male singing," in Proc. of Interspeech
2009,pp. 832-835,2009.
[92] D. Ruinskiy and Y. Lavner, "An effective algorithm for au­
tomatic detection and exact demarcation of breath sounds
in speech and song signals," IEEE Transactions on Audio,
Speech, and L anguage Processing, vol. 15, pp. 838-850,
2007.
[93] T. Nakano, J. Ogata, M, Goto and Y. Hiraga, "Analysis
and automatic detection of breath sounds in unaccompanied
singing voice," in Froc. of ICMPC 2008, pp. 387-390, 2008.
[94] p, Lal, "A comparison of singing evaluation algorithms," inProc. of Interspeech 2006, pp, 2298-2301,2006,[95] T. Nakano, M. Goto and Y. Hiraga, "An automatic singing
skill evaluation method for unknown melodies using pitch in­terval accuracy and vibrato features," in Proc. of Interspeech
200�pp. 1706-1709,2006.
[96] P. Prasert,K. Iwano and S. Furui,"An automatic singing voiceevaluation method for voice training systems," in Proceedingof the 2008 Spring Meeting of the Acoustical Society ofJapan,
pp. 911-912, 2008.
[97] R. Daido, S. Hahm, Ito, S. Makino and A. Ito, "A system forevaluating singing enthusiasm for karaoke," in Proc. ofiSMIR
2011,pp. 31-36,2011.
[98] D. M, Howard and G, F. Welch, "Microcomputer-based
singing ability assessment and development," Applied Acous­
tics, vol. 27, pp. 89-102, 1989.
[99] D. Hoppe, M. Sadakata and P. Desain, "Development of real­time visual feedback assistance in singing training: a review,"
Journal of Computer Assisted L earning, vol. 22,pp. 308-316,
2006.
[100] T. Nakano, M. Goto and y, Hiraga, "MiruSinger: A singing
skill visualization interface using real-time feedback and mu­
sic CD recordings as referential data," in Proc. of ISM 2007
Workshops (D emonstrations), pp. 75-76, 2007.
[101] D. Deutsch, ed. , The Psychology of Music. Academic Press,
1982.
[102] I. R. Titze,Principles of Voice Production. The National Cen­
ter for Voice and Speech, 2000.
[103] F. Husler and Y. Rodd-Marling, Singing: The Physical Natureof the Vocal Organ. A Guide to the Unlocking of the Singing
Voice. Hutchinson & Co, 1965.