INTERSPEECH 2006 - ICSLPAn Automatic Singing Skill Evaluation Method for Unknown Melodies
Using Pitch Interval Accuracy and Vibrato Features
Tomoyasu Nakano

Masataka Goto

Yuzuru Hiraga

Graduate School of Library,
Information and Media Studies,
University of Tsukuba, Japan

National Institute ofAdvanced Industrial Scienceand Technology (AIST), Japan

Graduate School of Library,
Information and Media Studies,
University of Tsukuba, Japannakano@slis.tsukuba.ac.jp

m.goto@aist.go.jp

hiraga@slis.tsukuba.ac.jp

Abstract

2. Acoustic Features for Automatic Singing
Skill Evaluation

This paper presents a method of evaluating singing skills that does
not require score information of the sung melody. This requires
an approach that is different from existing systems, such as thosecurrently used for Karaoke systems. Previous research on singing
evaluation has focused on analyzing the characteristics of singing
voice, but were not aimed at developing an automatic evaluation
method. The approach presented in this study uses pitch interval accuracy and vibrato as acoustic features which are independent from specific characteristics of the singer or melody. The approach was tested by a 2-class (good/poor) classification test with
600 song sequences, and achieved an average classification rate of
83.5%.
Index Terms: singing skill, automatic evaluation, unknown
melodies.

Human subjects can be seen to consistently evaluate the singing
skills for unknown melodies [8]. This suggests that their evaluation utilizes easily discernible features which are independent of
the particular singer or melody. Within the scope of this paper, we
propose pitch interval accuracy and vibrato as such feature candidates.
The proposed features are numbered and shown in boxed
form, like n . Throughout the paper, the singing samples are solo
vocal and are digital recordings of 16bit/16kHz/monaural.
2.1. Estimating Pitch Interval Accuracy
The pitch interval accuracy is judged by the fitting of the F0 (fundamental frequency) trajectory to a semitone (100 cent) width grid(corresponding to equal temperament in the Western Music Tradition). Hereafter, pitch/frequency values will be referred to bycents, which are log-scale frequency values. The cent value fcentof frequency fHz given as follows (middle C corresponds to 4800
cent).
fHz(1)
fcent = 1200 log2
3
440 × 2 12 −5
Suppose the semitone grid borders are set at multiples of 100.
Then a particular pitch x has an offset of F (0 ≤ F < 100) fromits nearest lower border. If F has a constant value throughout thesong sequence, then the singing can be seen to have a good fitting
to the semitone grid.
Let p(x; F ) be a Gaussian comb filter for pitch x and offset
F defined as follows, where ωi is a weight factor (currently set to1), and σi (= 16 cent) is the standard deviation of the Gaussian
distribution.
ﬀj∞X(x − F − 100i)2
ωi
√
(2)
exp −
p(x; F ) =
2σi2
2πσi
i=0

1. Introduction
The aim of this study is to explore a method of automatic evaluation of singing skills without score information. Our interest lies
in identifying the criteria that human subjects use in judging thequality of singing for unknown melodies, using acoustic features
which are independent from specific characteristics of the singer
or melody. Such evaluation systems can be useful tools for improving singing skills, and also can be applied to broadening thescope of music information retrieval and singing voice synthesis.
Previous work related to singing skills include those based ona control model of fundamental frequency (F0) trajectory [1], general characteristics [2, 3], as well as work on automatic discrimination of singing and speaking voices [4], and acoustic differences
between trained and untrained singers’ voices [5, 6, 7]. None ofthese work have gone as far as presenting an automatic evaluation
method.
This paper presents a singing skill evaluation scheme based on
pitch interval accuracy and vibrato, which are regarded as featuresthat function independently from the individual characteristics ofsinger or melody. To test the validity of these features, an experiment of automatic evaluation of singing performance by a 2-classclassification (good/poor) was conducted.
The following sections describe our approach and the experimental results of its evaluation. Section 2 presents discussion offeatures. Section 3 describes the classification experiment and its
evaluation. Section 4 concludes the paper, with discussion on directions for future work.

Using this as a filter function, the semitone stability Pg (F, t) attime t and width Tg is defined as follows, where FF 0 (t) denotes
the F0 and PF 0 (t) denotes the F0 possibility at time t, which are
estimated per 10 msec using the method of Goto et al. [9].
Z t
Pg (F, t) =p(FF 0 (τ ); F )PF 0 (τ )dτ
(3)
t−Tg

If all the F0 values are quantized by units of 100 cent, Pg (F, t)will have a single sharp peak at grid frequency Fg . The sharpness

1706

September 17-21, Pittsburgh, Pennsylvania

INTERSPEECH 2006 - ICSLP

Original F0 trajectory[cent]of the Pg (F, t) distribution thus indicates the degree of deviationof the singer from any semitone grid.
Figure 1 shows the calculation process and example results ofPg (F, t). The top figure shows the original F0 trajectory. The second figure shows the result of smoothing by an FIR lowpass filter
with 5 Hz cutoff frequency1 , and then removing the silent sections. The purpose of lowpass filtering is to remove the F0 fluctuations (overshoot, vibrato, and preparation) of the singing voice [1].
Pg (F, t) is calculated with a 200-sample (2sec) rectangular window shifted by 5 samples. The bottom left figures show example
fitting to grids with F = 19 and F = 78, which are cumulated as
Pg (F, t) values shown to the right.
Figure 2 shows examples of Pg (F, t) and its long-term average g(F ). When the singing is “good”, then Pg (F, t), and consequently, its g(F ) always has a single sharp peak. So the sharpnessof g(F ) can be utilized as a measure of pitch interval accuracy. Its
second moment M (defined as follows) is used as feature 1 :Z Fg +50(Fg − F )2 g(F )dF
(4)
M=

6000
5800
5600
5400
5200
100

50

150

250

300

350

400

450

[x 10msec][cent] F0 trajectory without silent sections after lowpass filtering
6000
5800
5600
5400
5200

50

100

150

200

250

300

[x 10msec]

Window-Shift[cent]
100

F = 78 cent

[cent]
5800
5700
5600
5500
5400
5300

200

100

...

p(x; F)[x 10msec]

F = 19 cent

[cent]
5800
5700
5600

Fg −50

5500
5400
5300

where Fg is the F value for maximum g(F ), i.e.:

100

Fg = argmax g(F )

200

0

200

p(x; F)[x 10msec](5)

Semitone stability Pg(F,t)

F

Figure 1: Overview of calculation method of Pg (F, t).

The second feature 2 is taken as the slope bg of the linear
regression line of function G(F ):g(Fg + F ) + g(Fg − F )G(F ) =
2
bg is obtained by minimizing
Z 50
errg2 =(G(F ) − (ag + bg F ))2 dF[cent][cent]

Pg(F,t) in good singing

100

100

g(F)

(6)
0

(7)

10

20

30

[cent]

0

40

50

60

70

80

90

100
[frames][cent]

Pg(F,t) in poor singing

100

0

100

g(F)

over ag and bg .
2.2. Estimating Vibrato Sections

0

Vibrato (deliberate, periodic fluctuation of F0) is considered asimportant singing technique, and so is incorporated within ourscheme. Our vibrato detection scheme imposes restrictions on vibrato parameters of rate (the number of vibrations per second) andextent (the amplitude of vibration from an average pitch on thevibrato section). Restrictions of rate and extent are based on previous research [2, 10], with the rate range set at 5–8 Hz and extent
range at 30–150 cent.
The basic idea is to detect vibrato by using short-term Fouriertransform (STFT). In our current implementation, an STFT with a
32-sample (320 msec) Hanning window is calculated by using theFast Fourier Transform (FFT). STFT is applied to ΔFF 0 (t), i.e.
the first order finite differential of FF 0 (t). The power spectrum
X(f, t) can be expected to have a sharp peak where f corresponds
to the vibrato rate. This is expressed by the power Ψv (t) and the
sharpness Sv (t) defined as:
Z FH
Ψv (t) =X̂(f, t)df
(8)

20

30

40

50

60

70

80

90 100[frames]0

Figure 2: Examples of Pg (F, t) and its long-term average g(F ) in
good/poor singing.
where (FL , FH ) is the range of vibrato rate, and X̂(f, t) is
X(f, t) normalized over f :X̂(f, t) = R

X(f, t)X(f, t)df(10)

Using these, the vibrato likeliness Pv (t) is defined as
Pv (t) = Sv (t)Ψv (t)(11)

A section is judged as a vibrato section when it has high valuesof Pv (t), and FF 0 (t) crosses its average value more than 5 times.
The vibrato rate and extent are obtained byFL˛
Z FH ˛˛˛˛ ∂ X̂(f, t) ˛
Sv (t) =
˛ df
˛
∂f ˛
FL ˛

10

N
1 X
1
Rn
=
·
rate
N n=1

(9)

1 We avoid unnatural smoothing by ignoring silence sections and leaps

extent =of F0 transitions wider than a 300 cent threshold.

1707

N
1 X
En
·
2N n=1(12)(13)

INTERSPEECH 2006 - ICSLP

Vibrato (deliberate, periodic fluctuation of F0 [[cent]

R1

R3

R2

])

Table 1: Dataset for classification experiment.

R4

5600

E3

E4

E2

E1

5500

E6

E7
average

R6

R5

5400
140

E5

170

160

150

R7
180

190

200

[x 10msec]

Figure 3: Extraction parameters for vibrato detection.
[cent]
6000

F0 trajectory () with estimated vibrato sections (

)

5600
5200

200

100

300

name

class

languagegender

E004
E008
E017
E021
J002
J054
E001
E002
E013
E023
J014
J052

good
good
good
good
good
good
poor
poor
poor
poor
poor
poor

EnglishEnglishEnglishEnglishJapaneseJapanese
EnglishEnglishEnglishEnglishJapaneseJapanesefemalefemalemale
male
femalemale
femalefemalemale
male
femalemalethe numberof samples
50
50
50
50
50
50
50
50
50
50
50
50

400 [x 10msec]

Vibrato likeliness Pv(t)3.2. Experimental Setting

0.2

Two experiments with different evaluation criteria were conducted
over three dataset settings (male, female, and male-female). In
each case, the feature value space was classified using Support
Vector Machine (SVM) as the classifier. The two evaluation criteria are 10-fold cross-validation and leave-one-out cross-validation.
The 10-fold cross-validation method uses 9/10 of the samples as
the training set and 1/10 as the test set in each trial. The leave-oneout cross-validation, in our context, evaluates one sample as test
data, and uses the rest as training data (excluding those with thesame singer/melody as the test data).

0.1
0

300

200

100

400 [x 10msec]

Figure 4: An example of vibrato detection.

where Rn [sec] and En [cent] are illustrated in Figure 3. The total
length of the vibrato section is used as feature 3 .
The following two functions Pv1 and Pv2 are also used as features 4 and 5 , since they take high values in the presence ofvibratos (Tv is the length of the analyzed signal).
Pv1 = max (Pv (t))
0≤t≤Tv

Pv2 =

1
Tv

3.3. Results and Discussion
Table 2 and Table 3 show the classification rates, precision rates
and recall rates of each class for the 10-fold and the leave-one-out
cross-validation respectively. The classification rate (C), precision
rate (Pi ) and the recall rate (Ri ) are defined as follows, where idenotes the class of good (i = good) or poor (i = poor).

(14)

Z Tv
Pv (t)dt

(15)

0

Pi =

3. Classification Experiment

samples correctly classiﬁed as classi
× 100
samples classiﬁed as classi(16)

samples correctly classiﬁed as classi
× 100
(17)
samples in classi
samples correctly classiﬁed
Rgood + Rpoor
C=
× 100 =
total number of samples
2
(18)
Ri =

The proposed features 1 – 5 have been tested in 2-class classification (good/poor) experiments, using the results of a previous
rating experiment by human subjects [8].
3.1. Dataset

The results for male and male-female datasets are similar inboth criteria, while there is a significant drop of ratings in theleave-one-out criteria for the female dataset. The female dataset
also has a slightly lower rating in the 10-fold criteria as well. The
reason for this drop is yet unclear, although the overall agreement in the male-female dataset suggests that the proposed features are both effective, and also robust against individual difference of singer and/or melody.
The overall high values of Pgood and Rpoor suggest that theclassification is stringent towards judging as “good” (if judged as
good, then it is likely to be “real good”).
Figure 5 shows the classification rate for singers from the
male-female dataset. The results show that the classification ofgood samples have a relatively high number of errors. One reason

The song samples are taken from the AIST Humming Database(AIST-HDB) [11]. The AIST-HDB contains singing voices of 100
subjects who sung melodies of 100 excerpts from 50 songs in the
RWC Music Database (Popular Music [12] and Music Genre [13]).
Table 1 shows the singing voice dataset used in the experiment. Our automatic classification experiment used 600 samples
by 12 singers who sung 50 excerpts from either 25 Japanese or
English songs, after listening to each excerpt five times. The 12
singer subjects (ID’s in the “name” column) were selected by thecriteria of receiving a consistently high rating (“good”) or low rating (“poor”) in the rating experiment by human subjects, and alltheir samples were marked good/poor accordingly (given in the“class” column in the table).

1708

INTERSPEECH 2006 - ICSLP

Pgood
95.2%
91.7%
90.3%

Rgood
79.3%
66.7%
74.7%

Ppoor
82.3%
73.8%
78.4%

Rpoor
96.0%
94.0%
92.0%

50

50

0

0

E0

Table 3: Results of the leave-one-out criteria.
Datasetmale
femalemale-female

C
87.7%
71.7%
83.5%

Pgood
93.8%
74.8%
87.6%

Rgood
80.7%
65.3%
78.0%

Ppoor
70.8%
58.0%
70.3%

Singer No.

Rpoor
94.7%
78.0%
89.0%

Poor

100

02
E0
13
E0
23
J0
14
J0
52

C
87.7%
80.3%
83.3%

04
E0
08
E0
17
E0
21
J0
02
J0
54

male
femalemale-female

[%]

E0
01

Dataset

Good

100

E0

[%]

Table 2: Results of the 10-fold criteria.

Singer No.

Figure 5: Result of the classification experiment (by singer) using
the leave-one-out criteria and the male-female dataset.

ysis, Modification and Synthesis System”, IPSJ Journal,
43(2):208–218, 2002. (in Japanese)is that even a good singer may occasionally fail to keep good pitchintervals, as they were singing out from memory. Such cases include when the overall pitch undergoes a gradual drift (resulting in
bad ratings in 1 2 ), which to the human ear does not sound so
distorted.
Classification errors also arise from the vibrato features 3 –[4] Ohishi, Y., Goto, M., Ito, K. and Takeda, K., “Discrimination
between Singing and Speaking Voices”, in Proc. 9th European Conference on Speech Communication and Technology(Interspeech2005), 1141–1144, 2005.
[5] Omori, K., Kacker, A., Carroll, L.M., Riley, W.D. and Blaugrund, S.M., “Singing Power Ratio: Quantitative Evaluationof Singing Voice Quality”, Journal of Voice, 10(3):228–235,
1996.

5 . A “good” sample can be misclassified as poor when there
is no vibrato, or when the values surpass the range restriction ofvibrato parameters. On the other hand, poor singing which cannot
keep a stable F0 can be mistakenly judged as a vibrato, especiallyresulting from relatively high values of 4 5 .

[6] Brown, W. S. Jr., Rothman, H. B. and Sapienza, C.M., “Perceptual and Acoustic Study of Professionally Trained Versus
Untrained Voices”, Journal of Voice, 14(3):301–309, 2000.

4. Conclusion[7] Watts, C., Barnes-Burroughs, K., Estis, J. and Blanton,D., “The Singing Power Ratio as an Objective Measure ofSinging Voice Quality in Untrained Talented and Nontalented Singers”, Journal of Voice, 20(1):82–88, 2006.

This paper proposed two acoustical features, pitch interval accuracy and vibrato, which are effective for evaluating singing skills
without score information. In addition to these features, we havealso investigated other features, such as the slope of long-term average spectrum (LTAS), the power within a frequency range of
singer’s formant, the variance of 16-order cepstral coefficients, theratio between the power of harmonic components and others, theaverage of spectral centroid or spectral rolloff, and the standard
deviation of F0 or power. The performance of their combinations,however, has not surpassed the performance of the two features
presented in this paper. In the future, we plan to investigate otherfeatures relevant to different vocal aspects such as rhythm and vocal quality.

[8] Nakano, T., Goto, M. and Hiraga, Y., “Subjective Evaluation of Common Singing Skills Using the Rank Ordering
Method”, in Proc. 9th International Conference of Music
Perception and Cognition (ICMPC2006), 2006. (accepted)[9] Goto, M., Itou, K. and Hayamizu, S., “A Real-time Filled
Pause Detection System for Spontaneous Speech Recognition”, in Proc. 6th European Conference on Speech Communication and Technology (Eurospeech ’99), 227–230, 1999.
[10] Seashore, C.E., “A Musical Ornament, the Vibrato”, Chapter 4, in Psychology of Music, McGrew-Hill Book Company,
pp.33–52, 1938.

5. Acknowledgements[11] Goto, M. and Nishimura, T., “AIST Humming Database:Music Database for Singing Research”, The Special Interest Group Notes of IPSJ (MUS), 2005(82):7–12, 2005. (inJapanese)

Authors would like to thank Mr. Hirokazu Kameoka (the University of Tokyo) for his valuable discussions and Dr. Elias Pampalk(CREST/AIST) for proofreading an earlier version.

[12] Goto, M., Hashiguchi, H., Nishimura, T. and Oka, R.,“RWC Music Database: Popular, Classical, and Jazz Music
Databases”, in Proc. 3rd International Conference on Music
Information Retrieval (ISMIR 2002), 287–288, 2002.

6. References[1] Saitou, T., Unoki, M. and Akagi, M., “Development of an
F0 Control Model Based on F0 Dynamic Characteristics for
Singing-voice Synthesis”, Speech Communication, 46: 405–
417, 2005.

[13] Goto, M., Hashiguchi, H., Nishimura, T. and Oka, R., “RWC
Music Database: Music Genre Database and Musical Instrument Sound Database”, in Proc. 4th International Conference on Music Information Retrieval (ISMIR 2003), 229–
230, 2003.

[2] Sundberg, J., The Science of the Singing Voice, Northern Illinois Univ Press, 226p., 1987.
[3] Kawahara, H. and Katayose, H., “Scat Generation Research
Program Based on STRAIGHT, a High-quality Speech Anal-

1709