2009 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

October 18-21, 2009, New Paltz, NY

A NOVEL FRAMEWORK FOR RECOGNIZING PHONEMES OF SINGING VOICE IN
POLYPHONIC MUSIC
Hiromasa Fujihara,†,‡ Masataka Goto,† and Hiroshi G. Okuno‡†

National Institute of Advanced Industrial Science and Technology (AIST)
Tsukuba, Ibaraki 305-8568, Japan

Kyoto University
Sakyo-ku, Kyoto 606-8501, Japan

this method segregated the singing voice from polyphonic music
to reduce the negative inﬂuence of the accompaniment sound, it
is generally based on conventional speech recognition techniques.
That is, it extracts feature vectors that represent the spectral envelopes, such as Mel-frequency cepstral coefﬁcients (MFCCs),and calculates the likelihood of the feature vectors using the Gaussian mixture models (GMMs).
Our previous method, however, exhibits the following two
problems:
Problems of segregation The performance of singing voice segregation critically affects that of the recognition because the F0
estimation errors deteriorate segregation quality, and it cumulatively degrades recognition performance. Besides, the segregation
process discards the other components of the spectrum, which include important information about the background noise (accompaniment sound) such as the S/N ratios and the degree of vocal
distortion it causes.
Inaccurate estimation of spectral envelope Our previous
method estimated a spectral envelope from a given (segregated)harmonic structure and calculated the distance between estimatedenvelopes. However, the harmonic components are considered
to be points sampled from their original spectral envelopes at aninterval of F0 along the frequency axis, and the perfect reconstruction of the spectral envelope from the harmonic structure is
generally impossible. Therefore, it was difﬁcult to calculate thedistance of high-pitched sounds, such as a female singing voice.
Gruhne et. al. [4] have worked on a phoneme recognition
problem. They also segregated a vocal using a similar method
used in [1]. Several studies have been done on automatic synchronization between music and lyrics [5, 6, 7, 8]. These methods
usually use conventional speech recognition techniques or more
simpliﬁed techniques and try to improve performance by integrating other information such as characteristics of the target language
and structure of the music.
We propose a novel method for overcoming these problems.
Our method neither segregates a singing voice nor reconstructs
a spectral envelope from a single harmonic structure. Instead, it
expresses the observed spectrum of a singing voices ”as is“, bystochastically modeling the generation process of the spectrogramof a singing voice with accompaniment sounds. In the training
process, we develop a method that can estimate a more accurate
spectral envelope by estimating it from many harmonic structures.

ABSTRACT
A novel method is described that can be used to recognize thephoneme of a singing voice (vocal) in polyphonic music. Though
we focus on the voiced phoneme in this paper, this method is design to concurrently recognize other elements of a singing voicesuch as fundamental frequency and singer. Thus, this method is
considered to be a new framework for recognizing a singing voicein polyphonic music. Our method stochastically models a mixtureof a singing voice and other instrumental sounds without segregating the singing voice. It can also estimate a reliable spectral envelope by estimating it from many harmonic structures with various
fundamental frequencies (F0s). The results of phoneme recognition experiments with 10 popular-music songs by 6 singers showedthat our method improves the recognition accuracy by 8.7 points
and achieves a 20.0% decrease in error rate.
Index Terms— Singing voice, Phoneme recognition, Spectral
modeling, Mixture of experts
1. INTRODUCTION
Music is an important media content in both industrial andcultural aspects, and a singing voice (vocal) is one of the most important elements of music. Among the many other considerable
elements of a singing voice, we deemed the lyrics, voice quality(or name of a singer), and fundamental frequencies (F0s) to be
the most fundamental elements of a singing voice. To develop a
computer that can recognize a singing voice, we have been working on an automatic synchronization method for singing voice andlyrics [1], an automatic singer identiﬁcation method [2], and an
F0 estimation method for a singing voice [3]. As a stepping stone
to further develop these methods, we propose a novel framework
for recognizing a singing voice within polyphonic music, which isnamed the W-PST (Weighted-composition of Probabilistic Spectral Template) method. We also verify the effectiveness of W-PST
method by conducting a phoneme recognition experiment on condition that correct F0s are given. Although we mainly focused on
lyrics, we designed W-PST method to be applicable in recognizing
F0s and names of singers.
Research in automatic lyric recognition from polyphonic music is important because it can be directly applied to a new music information retrieval system. Even if perfect lyric recognition cannot be achieved, the basic techniques for recognizing thephonemes can be used in an automatic synchronization method
for lyrics and music. However, automatic recognition of lyrics (orphonemes) of a singing voice is more difﬁcult than that of speech
because a singing voice ﬂuctuates wildly due to the vibrato, a wide
range of fundamental frequencies (F0s), and the emotional expression of singers. Furthermore, other instrumental sounds, which
are usually accompanied with the singing voice, also degrade theperformance of lyric (or phoneme) recognition.
We previously developed a method for automatically synchronizing lyrics with the corresponding singing voice [1]. Although

2. W-PST METHOD FOR RECOGNIZING SINGINGVOICEAs shown in Figs. 1 (c) and (d), we assume that the observed
spectrum of polyphonic music is generated from a set of probability distributions, which is called the probabilistic spectral template. That is, the power of each spectral bin follows a probabilistic
distribution, and forms of the probabilistic distributions differ frombin to bin. Assuming the additivity of the power spectrum, we consider that probabilistic spectral templates are produced by adding
two different probabilistic spectral templates in a linear scale; aThis research was supported in part by CrestMuse, CREST, JST.

978-1-4244-3679-8/09/$25.00 ©2009 IEEE‡17

2009 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

r
e
w
o
P

g
o
L

October 18-21, 2009, New Paltz, NY

Vocal envelope templates

gv

. . .

Observed Spectrum
500

1000

2000

4000

(a) Vocal spectral template Y
Frequency (Hz)

⊕

v, f

r
e
w
o
P
g
o
L

r
e
w
o
P

g
o
L

Phoneme /a/

r
e
w
o
P

g
o
L

Harmonic filter

gn
500

1000

2000

Frequency (Hz)

500

1000
2000
Frequency (Hz)(c) Probabilistic spectral
template Y

4000

500

4000

1000

2000

Frequency (Hz)(d) Observed spectrum y( f )Noise spectral templaten, fFigure 1. Generation process of the observed spectrum. The probability values are indicated by the darkness. Note that the S/N ratio
can be controlled by the gain parameters, gv and gn .

+
500

1000

2000

• how to optimize gain parameters (volumes).
Our approaches to overcome these issues are as follows,• We assume the distributions of a probabilistic spectral template
as log-Gaussian distributions.
• We approximate that the sum of probability variables that follow the log-Gaussian distributions also follows the log-normal
distribution1 .
• We optimize the parameters using the quasi-Newton method.

r
e
w
o
P

g
o
L

500

4000

(a) Vocal envelope templateY

r
e
w
o
P

g
o
L

1000

2000

'

v, f

4000

500

1000

2000

4000

(b) Harmonic filter H ( f ; f 0 ) (c) Vocal spectral template Y
Frequency (Hz)Frequency (Hz)

Select the most likely template (phoneme)

Figure 3. Overview of phoneme estimation process.

f0

r
e
w
o
P
g
o
L

. . .

4000

f

(b) Noise spectral template Y

Phoneme /i/
Phoneme /e/Estimate optimal parameterparameters

Frequency (Hz)v, fFigure 2. Example of vocal spectral template, which generated
from the vocal envelope template and the harmonic ﬁlter.

3.1. Probabilistic spectral template
We assume a spectrum of polyphonic sound mixture, y(f ),is generated from the probabilistic variables Yf . We call thesevariables the probabilistic spectral template. Here, f represents
a frequency in log scale, and y represents a spectral power in log
scale.
We then assume that Yf can be divided into two different probabilistic spectral templates, Yv,f and Yn,f , by the following equation,
Yf =log(exp(Yv,f + gv ) + exp(Yn,f + gn )),(1)
where Yv,f represents the spectrum of a vocal, which is called the
vocal spectral template, and Yn,f represents that of other instrumental sounds, which is called the noise spectral template. Here,gv and gn represent the gain parameters. By changing gv and gn ,this model can be used to control the S/N ratio of the vocal andnoise spectral templates. Note that we assume the additivity of thepower spectrum in linear scale.
We assume that Yv,f and Yn,f follow the Gaussian distribution(in log scale) and are represented by
2),(2)
Yv,f ∼N (y; μv,f , σv,f2),
(3)
Yn,f ∼N (y; μn,f , σn,fwhere N (y; μ, σ 2 ) represents the Gaussian distribution with mean
μ and variance σ 2 .
Assuming the source-ﬁlter model, we consider the harmonicsound Yv,f (singing voice) to be a sum of the probabilistic modelof the envelope and a ﬁlter function, which represents the harmonic
structure in log scale (Fig 2).
Yv,f =Yv,f+ log H(f ; f0 )
(4)
2
)
(5)
∼N (y; μv,f + log H(f ; f0 ), σv,fX
2
H(f ; f0 )=N (f ; log f0 + log h, σH),(6)vocal spectral template (Fig. 1 (a)), which generates the spectrumof a singing voice, and a noise spectral template (Fig. 1 (b)), which
generates the spectrum of accompaniment sounds. By introducing
gain parameters for both templates and taking their weighted sum,
we can represent the spectrum of various S/N ratios and volumes.
A vocal spectral template is produced by multiplying a vocal envelope template (Fig. 2 (a)), which represents the spectral envelope,by a harmonic ﬁlter (Fig. 2 (b)), which represents the harmonic
structure. The shape of the harmonic ﬁlter is controlled by an F0parameter. The likelihood of observed spectrum for this probabilistic model can be calculated if the parameters of this model —the F0 of the harmonic ﬁlter and the gain parameters of the vocal
and noise spectral templates — are given. Thus, we can estimate aphoneme involved in observed spectrum by preparing vocal envelope templates of various phonemes and selecting the most likely
one (Fig. 3). We can also estimate the F0 of a singing voice by
estimating the most likely value.
The novelties of W-PST method are summarized as follows:• It does not segregate singing voice, but our framework recognizes a singing voice directly from polyphonic sound mixtures.
Since humans understand a singing voice without segregatingit, this process is natural from the standpoint of human perception.
• It is robust against ﬂuctuations in accompaniment sound because it estimates the S/N ratio of the moment. It becomes
more robust if we prepare various types of noise spectral templates and select the most likely one.
• It is robust against high-pitched sounds because it does not
estimate an envelope from a single harmonic structure.
• It is easy to integrate unvoiced consonants by only preparing new templates for these phonemes (without using the harmonic ﬁlter).

h2where Yv,f
∼ N (y; μv,f ; σv,f
) represents probabilistic variablesof the vocal envelope, which is called the vocal envelope template,and H(f ; f0 ) represents the ﬁlter of F0, f0 , which is called theharmonic ﬁlter. Note that the harmonic ﬁlter H(f ; f0 ) is not a
probabilistic variable.

3. FORMULATIONThis section describes a concrete formulation of W-PST
method described in Sec. 2. To implement it, we have to design amethod for overcoming the following three issues,• how to represent a probabilistic spectral template,• how to calculate the sum of two probabilistic spectral templates, and3.2. Approximation of sum of two spectral templates
Since a probabilistic spectral template Yf , which is expressedin (1), is difﬁcult to calculate, we approximate Yf using the Gaus1 The sum of probability variables that follow the log-Gaussian distributions does not follow a log-normal distribution in general.

18

2009 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

sian distribution. The ﬁrst-order Taylor series of the function
f (x1 , x2 ) = log(exp(x1 ) + exp(x2 )) at (μv,f + gv , μn,f + gn )can be denoted by linear combination of x1 and x2 . Since linear combination of probabilistic variables that follow the Gaussian distribution also follows the Gaussian distribution, thus, Yf =f (Yv,f , Yn,f ) can be approximated as follows.
(7)
Yf ∼N (y; μf , σf2 )μf =log(exp (μv,f + gv ) + exp (μn,f + gn ))(8)22(exp (μv,f + gv ))2 σv,f+ (exp (μn,f + gn ))2 σn,f
σf2 =
(9)
(exp (μv,f + gv ) + exp (μn,f + gn ))2

r
e
w
o
P

g
o
L

gv ,gn

r
e
w
o
P

g
o
L

500

2000

1000

3000

4000

500

Frequency (Hz)

1000

2000

3000

4000

2000

3000

4000

Frequency (Hz)(a) Initial value(b) t = 2

r
e
w
o
P
g
o
L

3.3. Phoneme recognition and F0 estimation
To recognize a phoneme by using this model, ﬁrst, an indivisual template, θvp , for each phoneme p has to be prepared. Given
the F0 of the singing voice f0 and the observed spectra y(f ), we
can estimate the phoneme, p̂, involved in the spectra by the following equation,Zp̂ = argmax max
log N (s(f ); uf , σf2 ),(10)p

October 18-21, 2009, New Paltz, NY

r
e
w
o
P
g
o
L

500

2000

1000

3000

4000

500

Frequency (Hz)1000
Frequency (Hz)(c) t = 20(d) Each expert (t = 20)

Figure 4. Example of parameter estimation of the mixture of experts (MoE), where t represents the number of iteration. The middle line represents the mean of the MoE and the two thin lines
around it represent the standard deviation. The background minutedots represent the harmonic components of the training data. The
gating functions, Gi (f |Φ), are drawn at the bottom.

fwhere uf and σf are deﬁned by (8) and (9), respectively. Although
we did not evaluate F0 estimation in the experiments in this paper, we can concurrently estimate F0 and a phoneme by taking theargmax of f0 . When determining individual singers, we prepare
an individual template for each singer.

2)
ψm N (f |μm , σm
.
(14)
2


ψ
N
(f
|μ
,
σ
m
m m
m )
2
2
} is a set of unknown parameters,
Here, {ψm , μm , σm , am , bm , βmP
where ψm satisﬁes ψm ≥ 0 and m ψm = 1. These parameters
can be estimated using the expectation-maximization (EM) algorithm.
2)= P
Gm (f |ψm , μm , σm3.4. Optimization using quasi-Newton method
We optimize the parameter θ = (gv , gn ) using the quasiNewton method based on the BFGS (Broyden-Fletcher-GoldfarbShanno) method, which is a class of hill-climbing optimization
techniques. Given the observed spectrum s(f ), the target function
to be minimized, Q(θ), becomes
as follows;Z
Q(θ) = − log N (s(f ); uf , σf2 ).
(11)

4.2. Iterative Parameter Estimation
When we observe harmonic structures si (i = 1, . . . , I),
which consist of the log power of the h-th harmonic component,
yi,h , and its frequency, fi,h , denoted bysi = {(fi,1 , yi,1 ), . . . (fi,h , yi,h ), . . . (fi,Hi , yi,Hi )},(15)
the target likelihood function to be maximized is deﬁned by
Hi
N X
X
L=
N (yi,h + ki ; μv,fi,h , σv,fi,h ),(16)f

4. ESTIMATION OF SPECTRAL ENVELOPEin
We estimate a spectral envelope that is represented by Yv,f(4). The spectral envelope of the singing voice cannot be directly
observed because what we can observe is a harmonic structure
which is considered to be points sampled from its original spectral envelope. Thus, it is difﬁcult to estimate its original spectralenvelope from a single harmonic structure in general. We overcome this difﬁculty and estimate a reliable spectral envelope using many harmonic structures with various F0s. Moreover, since
we estimate the spectral envelope as a set of probabilistic distributions, the estimated envelope is robust against the ﬂuctuation of thesinging voice and the difference in conditions between the training
and testing data.
Since volumes could differ from frame to frame, we need ascheme to normalize such volume differences when we estimate
the envelope from many harmonic structures. We consider thevolume of each frame as an unknown parameter, and estimate it
concurrently with the parameters of the model for estimating thespectral envelope.

ihwhere ki represents the offset parameter, which normalizes thevolume of the harmonic structure. Since it is difﬁcult to estimate
both ki and the parameters of MoE at the same time, we updatethem sequentially.
The procedure of parameter estimation is summarized as follows.
Step 0 Set ki = 0 and initialize the other parameters.
Step 1 Update parameters of the MoE using the EM algorithm.
Step 2 Update ki using the following equation,
PHi μv,fi,h −yi,n
ki =

h=1

PHi

2
σv,f

i,h

1

(17)

h=1 σ 2v,fi,hStep 3 Go back to step 1.
Figure 4 shows an example of the parameter estimation process.
As for the noise spectral envelope, we can estimate the parameter in the same manner by considering the spectrum as si (i =1, . . . , I).

4.1. Mixture of Experts
For a spectral template model, we use the mixture of experts(MoE) model [9] based on the linear regression model. This model2
represents μv,f and σv,fof a spectral template asX2
Gm (f |ψm , μm , σm)(am f + bm )(12)
μv,f =
i
X
2
2 2 2
σv,f
=
Gi (f |ψm , μm , σm
) βm ,(13)

5. EXPERIMENTS
We conducted experiments on phoneme recognition by using
10 Japanese songs performed by 6 singers (3 male, 3 female) takenfrom the “RWC Music Database: Popular Music” (RWC-MDB-P2001)[11]. The target phonemes were 5 Japanese vowels; “a”, “i”,

i2
where Gm (f |ψm , μm , σm
) is the output of the gating network,
and we use a normalized Gaussian function [10] deﬁned by19

2009 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

October 18-21, 2009, New Paltz, NY

Table 1. Experimental results (%).
(i)
(ii)
(iii)
Gender∗∗∗ Singer Baseline 1 Baseline 2 Proposed
No. 4
M
A
31.1∗∗
33.0∗∗
64.3
No. 11
M
A
52.0
57.1
63.0
No. 9
M
B
30.0∗∗
48.4
52.6∗∗
No. 12
M
B
33.8
67.5
69.3
No. 15MC
42.6∗∗
50.8
61.7
No. 2
F
D
59.1
70.7
70.7
No. 16
F
D
57.2
63.1
69.9
No. 7
F
E
54.4
62.3
70.2
No. 18
F
E
59.0
66.9
71.6
No. 14
F
F
40.4
43.9
46.2
Average
46.0
56.4
65.1
∗ Song number of RWC-MDB-P-2001[11].
∗∗ Songs of which the model of the difference gender was selected.
∗∗∗ M and F denotes male and female, respectively.

6. DISCUSSION AND CONCLUSIONS
We have developed the W-PST method for recognizing thephonemes of a singing voice in polyphonic music. Although weevaluated this method through only phoneme recognition experiments, it is directly applicable to F0 estimation and singer identiﬁcation tasks. Thus, W-PST method can be considered as a
new framework for recognizing singing voice in polyphonic music. The underlying idea of this framework is that humans do notsegregate the target sound when listening to it. Unlike the typicalframework for polyphonic sound recognition, which ﬁrst segregates the target and then recognizes it, our framework recognizes asinging voice directly from a polyphonic sound mixture. This policy takes advantage of the characteristics of noise to improve theperformance, which is generally discarded in conventional methods.
The W-PST method has a commonality with the parallelmodel combination (PMC) technique [12] used in research of automatic speech recognition, which decompose noisy speech hidden Markov models (HMMs) from clean speech and noise HMMs.
The advantage of our method is that it estimates the S/N ratio of a
vocal and noise at every frame, while the PMC technique is used
to decompose HMMs on a ﬁxed S/N ratio beforehand.
Our goal is to develop an automatic lyrics recognition system.
To achieve this goal, we will expand our method to achieve higher
performance. For example, we plan to introduce a 3-D templatethat consists of not one frame but a number of consecutive frames
to represent the dynamic features of a singing voice. This idea ispromising because many researchers have proven the importanceof the delta features for speech recognition.

Song #∗“u”, “e”, and “o”. We conducted 6 fold cross validation, that is,when we evaluated a song of a singer, the vocal and noise templates (we call them phoneme model) were trained using songs ofthe other 5 singers. We used a manually annotated phoneme labelof the songs for both training data and ground-truth. Accuracy isdeﬁned as the ratio of the number of frames that are correctly estimated to the total number of frames. Only frames that involve
the target 5 vowels are used for calculating the accuracy. We used
the gender-dependent phoneme models here, that is, we trained thephoneme models for male and female separately. When testing, we
calculate a likelihood for each phoneme model and select a resultof the most likely model.
We tested our method under the following 3 conditions.
(i) Baseline 1 Extract MFCCs, ΔMFCCs, and ΔPower frompolyphonic audio signals and recognize them using the GMMs.
(ii) Baseline 2 Use the feature extraction method used in [1]; segregate the singing voice based on the harmonic structure before
extracting MFCCs, ΔMFCCs, and ΔPower and recognize themusing the GMMs.
(iii) Proposed Use the W-PST method proposed in this paper.
Note that manually-annotated F0s of the singing voice were used
for conditions (ii) and (iii) for both training and testing.
In condition (iii), we used the wavelet transform with the Gabor wavelet for spectrum analysis, and set the number of mixtureof the MoE to 10. The vocal and noise templates were created by
combining the templates that are trained using each of the songs
used for training. Therefore, there were a number of templates foreach phoneme in the templates. The vocal templates were trainedusing the vocal-only tracks of the songs, and the noise template
was trained using a karaoke (without vocal) track of the songs. In
condition (i) and (ii), we used the Short Time Fourier Transform
for spectrum analysis, and set the number of mixtures of GMMs
and the number of dimensions of MFCCs to 12 and 32, respectively. As for condition (ii), the data used for training GMMs werealso segregated based on the harmonic structures.
The results are summarized in Table 1. We can see that our
method increased the average accuracy by 8.7 points compared to
the result of condition (ii). We can also see that there was no songof which the accuracy was decreased. As shown in Table 1, ourmethod was able to select the model of the proper gender for all
songs, while baseline methods failed to select the model of theproper gender for some songs. Inspection of the incorrect frameswith our method (condition (iii)) and the correct frames with thebaseline method (condition (ii)) showed that 52.6% of incorrectframes of our method was correctly recognized using the baseline
method. By this fact, we can say that there is a possibility to further
improve performance by combining our method with the baseline
method.

7. REFERENCES[1] H. Fujihara, M. Goto, J. Ogata, K. Komatani, T. Ogata, and H. G.
Okuno, “Automatic synchronization between lyrics and music CD
recordings based on viterbi alignment of segregated vocal signals,” inProc. ISM, 2006, pp. 257–264.
[2] H. Fujihara, T. Kitahara, M. Goto, K. Komatani, T. Ogata, and H. G.
Okuno, “Singer identiﬁcation based on accompaniment sound reduction and reliable frame selection,” in Proc. ISMIR, 2005, pp. 329–336.
[3] ——, “F0 estimation method for singing voice in polyphonic audio
signal based on statistical vocal model and Viterbi search,” in Proc.
ICASSP, 2006, pp. 253–256.
[4] M. Gruhne, K. Schmidt, and C. Dittmar, “Phoneme recognition in popular music,” in Proc. ISMIR, 2007, pp. 369–370.
[5] K. Chen, S. Gao, Y. Zhu, and Q. Sun, “Popular song and lyrics synchronization and its application to music information retrieval,” in Proc.
MMCN’06, 2006.
[6] D. Iskandar, Y. Wang, M.-Y. Kan, and H. Li, “Syllabic level automaticsynchronization of music signals and text lyrics,” in Proc. ACM Multimedia, 2006, pp. 659–662.
[7] C. H. Wong, W. M. Szeto, and K. H. Wong, “Automatic lyrics alignment for cantonese popular music,” Multimedia Syst., vol. 4-5, no. 12,
pp. 307–323, 2007.
[8] M.-Y. Kan, Y. Wang, D. Iskandar, T. L. Nwe, and A. Shenoy, “Lyrically: Automatic synchronization of textual lyrics to acoustic musicsignals,” IEEE Trans. Audio, Speech, and Language Process., vol. 16,
no. 2, pp. 338–349, 2008.
[9] R. J. Jacobs, M. Jordan, S. J. Nowlan, and G. E. Hinton, “Adaptive mixtures of local experts,” Neural Computation, vol. 3, pp. 79–87, 1991.
[10] L. Xu, M. I. Jordan, and G. E. Hinton, “An alternative model for mixtures of experts,” Adv. Neural Inf. Process. Syst. 7, pp. 633–640, 1994.
[11] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka, “RWC Music
Database: Popular, classical, and jazz music databases,” in Proc. ISMIR, 2002, pp. 287–288.
[12] M. J. F. Gales and S. Yound, “An improved approach to the hidden
Markov model decomposition of speech and noise,” in Proc. ICASSP,
1997, pp. 835–838.

20