Alma Mater Studiorum University of Bologna, August 22-26 2006

Subjective evaluation of common singing skills
using the rank ordering method
Tomoyasu Nakano

Masataka Goto

Yuzuru Hiraga

Graduate School of Library,
Information and Media Studies,
University of Tsukuba
Tsukuba, Ibaraki 305-8550, Japannakano@slis.tsukuba.ac.jp

National Institute ofAdvanced Industrial Scienceand Technology (AIST)
Tsukuba, Ibaraki 305-8568, Japanm.goto@aist.go.jp

Graduate School of Library,
Information and Media Studies,
University of Tsukuba
Tsukuba, Ibaraki 305-8550, Japanhiraga@slis.tsukuba.ac.jp

Keywords

ABSTRACTThis paper presents the results of two experiments on singing skill evaluation, where human subjects judge the subjective quality of previously unheard melodies. The aim of
this study is to explore the criteria that human subjects use
in judging singing skill and the stability of their judgments,
as a basis for developing an automatic singing skill evaluation scheme.
The experiments use the rank ordering method, where thesubjects ordered a group of given stimuli according to theirpreferred rankings. Experiment 1 uses real, a capella singing as the stimuli, while experiment 2 uses the fundamental
frequency (F0) sequence extracted from the singing. In
experiment 1, 88.9% of the correlation between the subjects' evaluations was significant at the 5% level. Results ofexperiment 2 show that the F0 sequence is significant in
only certain cases, so that the judgment and its stability inexperiment 1 should be attributed to other factors of real
singing.

singing skill, subjective evaluation, rank ordering method

BACKGROUND
Automatic evaluation of singing skills is a promising research topic with various applications in scope. Previous
research on singing evaluation has focused on trained, professional singers (mostly in classic music), using various
approaches from physiology, anatomy, acoustics, and psychology – with the aim of presenting objective, quantitative
measures of singing quality. Such works have reported thatthe singing voices have singer's formant [1] and the specific characteristics of fundamental frequency (F0) [2]. In
particular, the singer's formant characterizes singing quality
as ringing [3].
Our interest is directed more towards ordinary, common
person's singing, understanding how they mutually evaluate
their quality, and to incorporate such findings in an automatic evaluation scheme.

AIMSThe aim of this study is to explore the criteria that humansubjects use in judging singing skill, and identify whether
such judgments are stable and in mutual agreement amongdifferent subjects. This will serve as a preliminary basis forour goal of developing an automatic singing skill evaluation scheme.
Two experiments were carried out. Experiment 1 is intended to verify the stability of human judgment, using acapella singing sequences (solo singing) as the stimuli.
Experiment 2 uses the F0 sequences (F0 singing) extracted
from solo singing, and is intended to identify their contribution in the judgment. In both experiments, the melodies
were previously unheard by the subjects.

In: M. Baroni, A. R. Addessi, R. Caterina, M. Costa (2006) Proceedingsof the 9th International Conference on Music Perception & Cognition(ICMPC9), Bologna/Italy, August 22-26 2006.©2006 The Society forMusic Perception & Cognition (SMPC) and European Society for theCognitive Sciences of Music (ESCOM). Copyright of the content of anindividual paper is held by the primary (first-named) author of that paper. All rights reserved. No paper from this proceedings may be reproduced or transmitted in any form or by any means, electronic ormechanical, including photocopying, recording, or by any information
retrieval systems, without permission in writing from the paper's primaryauthor. No other part of this proceedings may be reproduced or transmitted in any form or by any means, electronic or mechanical, including
photocopying, recording, or by any information retrieval system, without
permission in writing from SMPC and ESCOM.

ISBN 88-7395-155-4 © 2006 ICMPC

1507

ICMPC9 Proceedings

tion ρ [5] defined as follows:

METHOD AND EXPERIMENTS
The standard method of subjective evaluation by giving
grade scores to each tested stimuli [4] is inappropriate forour case of singing evaluation, where the subtleties of subjects' judgments may be obscured by differences in musicalexperience. So instead, we used a rank ordering method,where the subjects were asked to order a group of stimuli
according to their preferred rankings.
The singing samples are digital recordings of
16bit/16kHz/monaural. In order to suppress the variance
between the samples, all the samples were set at the same
volume and were presented through a headphone.

ρ = 1−

N6(ai − bi )2∑
3
N − N i =1(1)where N is the number of stimuli (= 10 in the experiments),and ai , bi are the i-th component (rank value) of the rank
vectors a and b. The value of ρ ranges from 1 (a = b) to − 1(a, b are reverse order). The correlation of a, b is significant at the 1% level for ρ ≥ 0.7333 and 5% level forρ ≥ 0.5636 [5].

Experiment 1
This experiment uses solo singing as the stimuli. The subjects were presented with four groups of singing, eachgroup with the same melody sung by 10 singers. The task
is to order each group using the interface explained above.
The subjects were free to listen to the melodies as manytimes as they want to.
The subjects were also asked to give introspective description of their judgments.

Interface for Subjective Evaluation
Figure 1 shows the interface screen used in the experiments.
The speaker icons indicate 10 stimuli (A, B, ..., J), which
can be double-clicked to play the sound, and can be moved
around by drag-and-drop using the mouse. The subjects are
instructed to align the icons horizontally according to theirorder of judgment, ranging from poor (left-hand side) togood (right-hand side). The vertical positioning is insignificant for the experiments.
The left figure shows an initial setting (random order), andthe right figure shows an example result, with H judged as
the best and B as the poorest. At the end of the experiment,
the subjects are also instructed to insert two lines (1. and 2.
in the right figure) classifying the samples into "good" (H,I), "poor" (B, F, D) and "intermediate" (E, J, A, G, C).

Subjects
22 subjects (University students, ages 19 to 29) participatedin the experiment. 16 had experience with musical instruments, and 2 had experience with vocal music (popular orchorus). 4 stated to possess absolute pitch. The subjects
were divided into two sets (A, B, each with 11 subjects),
each set presented with the same stimuli set.

Stimuli
The samples of stimuli were taken from the RWC Music
Database: Popular Music (RWC-MDB-P-2001) [7] andthe AIST Humming Database (AIST-HDB) [6]. The AISTHDB contains singing voices of 100 subjects, each singing
the melodies of two excerpts from the chorus and verse A
sections of 50 songs (100 samples) in the RWC Music Database (Popular Music [7] and Music Genre [8]) Table 1
shows the two stimuli sets A and B. Each set has 4 different melodies, sung by 10 individuals of the same gender (1from RWC-MDB-P and 9 from AIST-HDB) presented as agroup on the interface screen. The language of the lyrics iseither Japanese or English.

Figure 1. Example subjective evaluation session using
the interface screen..

The Measurement of Rank Correlation
The results are analyzed using the Spearman's rank correla-Experiment 2Experiment 2 follows the same procedure as experiment 1,
except that the stimuli are replaced with F0 singing, extracted from the solo singing used in experiment 1 (seebelow). The subjects were further instructed to ignore any
noise cased by the F0 extraction process.

Proceedings of the 9th International Conference on Music Perception &Cognition (ICMPC9). ©2006 The Society for Music Perception & Cognition (SMPC) and European Society for the Cognitive Sciences ofMusic (ESCOM). Copyright of the content of an individual paper is heldby the primary (first-named) author of that paper. All rights reserved. No
paper from this proceedings may be reproduced or transmitted in any
form or by any means, electronic or mechanical, including photocopying, recording, or by any information retrieval systems, without permission in writing from the paper's primary author. No other part of thisproceedings may be reproduced or transmitted in any form or by anymeans, electronic or mechanical, including photocopying, recording, orby any information retrieval system, with permission in writing from
SMPC and ESCOM.

ISBN 88-7395-155-4 © 2006 ICMPCSubjects
20 subjects (University students, ages 19 to 35) participatedin the experiment. None of them participated in experiment
1. 17 had experience with musical instruments, and 6 hadexperience with vocal music (popular or chorus). 6 stated1508

ICMPC9 Proceedings

to possess absolute pitch. The subjects were divided into
two sets (A, B, each with 10 subjects), each set presented
with the same stimuli set.

Stimuli
The stimuli used in this experiment are F0 sequences extracted from the samples used in experiment 1, removing
all other vocal features. F0 is estimated per 10 msec using
the method of Goto et al. [9], and is resynthesized as asinusoidal wave with its amplitude preserving the power ofthe most predominant harmonic structure of the original.
The resulting F0 sequence a natural impression comparable
to the original.
Table 1. 80 stimuli (by 40 singers).
setAB

music excerpted
No.
section27
verse Alyrics
languageJapanesemalethe numberof singers
1028

verse AJapanese

female1090

verse A

English

male

1097

chorusEnglishfemale1027chorus

Japanesemale

1028chorus

Japanesefemale1090

chorusEnglish

male

1097

verse A

Englishfemale10genderNote: music No. are from RWC-MDB-P-2001

RESULTSThe ranking correlation (1) is calculated for all pairs ofsubject rankings, giving the ρ-matrix shown schematicallyin Figure 2. In the top figure, I corresponds to pairings forrankings in experiment 1 (55 = 11 x (11 - 1)/2 pairs), II topairings in experiment 2 (45 = 10 x (10 - 1)/2 pairs), andIII to cross-pairings for rankings in experiments 1 and 2(110 = 11 x 10 pairs). The bottom figure shows an example
gradation display of the ρ-matrix for the " I -A-Englishfemale" case, corresponding to set A, English lyrics, female
singer group. The gradation is darker for higher ρ values.
Tables 2,3 show the results for I (ρ values of solo singing).
Table 2 shows the percentage of significant pairs, and Table 3 shows the statistical values of ρ for each of the groups.
The results show the stability of subject judgments for solosinging.
Each singing sample was further labeled as good, poor orotherwise (which is to be used for developing the automatic
evaluation scheme), using the following criteria:
good many subjects evaluated the sample as goodand no subject as poor,
many subjects evaluated the sample as poor
poor
and no subject as good,neither of the above.
otherwise
Table 4 shows the results of labeling.
Table 2. Percentage of significant pairs (solo singing).
setA

lyricslanguage
Japanese

gender

p<.01

p<.05

male

96.4% (53)100.0% (55)Japanesefemale

74.6% (41)

90.9% (50)Englishmale61.8% (34)

89.1% (49)Englishfemale80.0% (44)90.0%(198)
72.7% (40)Japanesemale

41.8% (23)
68.6%(151)
45.5% (25)Japanesefemale72.7% (40)

98.2% (54)Englishmale

52.7% (29)

89.1% (49)Englishfemale74.6% (41)
61.4%(135)
65.0%(260)

90.9% (50)
87.8%(193)
88.9%(391)overall (220)B

overall (220)overall (440)

Table 3. Statistics of ρ (solo singing).
setA
Figure 2. Graphical scheme of the ρ-matrix (above) and exISBNgradation
88-7395-155-4© for
2006theICMPC
1509
ampledisplay
A—English—female case.

lyricslanguage
Japanese

gendermean (SD)min / maxmale

0.87 (0.07)

0.71 / 0.99

ICMPC9 Proceedings

B

Japanesefemale0.77 (0.14)

0.38 / 0.95

Englishmale

0.75 (0.14)

EnglishfemaleJapaneseA

Japanese

male

0.68 (0.17)

0.22 / 0.94

0.28 / 0.96

Japanesefemale0.69 (0.18)

0.26 / 0.94

0.69 (0.14)

0.42 / 0.98

Englishmale

0.44 (0.27)-0.24 / 0.89

male

0.64 (0.22)

0.03 / 0.98

Englishfemale0.32 (0.39)-0.87 / 0.88

Japanesefemale0.81 (0.13)

0.39 / 0.99Japanesemale

0.27 (0.27)-0.33 / 0.79

Englishmale

0.73 (0.14)

0.36 / 0.98

Japanesefemale0.52 (0.23)-0.03 / 0.89

Englishfemale0.76 (0.14)

0.36 / 0.96

Englishmale

0.45 (0.29)-0.21 / 0.87

B

Englishfemale0.64 (0.16)
0.26 / 0.94
Table 7. Percentage of significant pairs(solo—F0 singing).

Table 4. Results of labeling (good/poor).
setAB

lyricslanguage
Japanese

set

A

p<.05

male

54.5% (60)

82.7% (91)Japanesefemale40.0% (44)78.2% (86)Englishmale

25.5% (28)

56.4% (62)English

femalemale

20.9% (23)
35.2%(155)
12.7% (14)55.5% (61)
68.2%(300)
27.3% (30)Japanesefemale29.1% (32)

60.0% (66)Englishmale

21.8% (24)

44.5% (49)Englishfemale41.8% (46)26.4%(116)
30.1%(271)

78.2% (86)52.5%(231)
60.3%(531)otherwise

male

3/10

2/10

5/10

Japanesefemale

3/10

3/10

4/10

English

male

4/10

2/10

4/10

Englishfemale

3/10

2/10

5/10

Japanese

male

1/10

3/10

7/10

Japanesefemale

3/10

3/10

4/10

Japanese

English

male

2/10

2/10

6/10

lyricslanguage
JapaneseAoverall (220)B

overall (220)gender

p<.01

p<.05

male

44.4% (20)

77.8% (35)Japanesefemale55.6% (25)

71.1% (32)Englishmale

15.6% (7)37.8% (17)Englishfemale35.6% (16)
55.6%(100)13.3% (6)overall (440)

Table 8. Statistics of ρ (solo—F0 singing).
set

lyricslanguage
Japanese

gendermean (SD)min / maxmale

0.72 (0.17)

0.24 / 0.98

Japanesefemale0.66 (0.18)

0.13 / 0.92

Englishmale

0.56 (0.24)-0.21 / 0.99

Japanesemale

17.8% (8)33.3%(60)
2.2% (1)Japanesefemale22.2% (10)44.4% (20)Englishmale

15.6% (7)46.7% (21)Englishfemale0.48 (0.32)-0.44 / 0.90

Englishfemale40.0% (18)
20.0%(36)
26.7%(96)62.2% (28)
41.7%(75)
48.6%(175)Japanese

male

0.42 (0.26)-0.27 / 0.90

Japanesefemale0.61 (0.19)

0.08 / 0.92

Englishmale

0.50 (0.24)-0.10 / 0.98

Englishfemale0.66 (0.17)

0.14 / 0.95

overall (180)overall (360)A

B

Table 6. Statistics of ρ (F0 singing).
setp<.01

poor

overall (180)B

gendergood

Englishfemale3/10
4/103/10
Table 5. Percentage of significant pairs (F0 singing).

setlyrics
language
Japanese

genderlyricslanguagegendermean (SD)ISBN 88-7395-155-4 © 2006 ICMPC

min / max

1510

ICMPC9 Proceedings

Tables 5,6 corresponds to the results of Tables 2,3 for II(F0 singing), and Tables 7,8 for III (solo—F0 singing crosscorrelation). The results for II show the stability of subject
judgments for F0 singing, while the results for III show thecorrelation between judgments for solo and F0 singing,indicating the amount of contribution of the F0 factor. Figure 3 shows the bar graph indicating that the results of Tables 2, 5, 7.

rhythmical stability•

pronunciation quality•

singing technique (e.g. vibrato, keeping a stable F0)•

vocal expression and quality•

good/poor can be classified from a short sequence(3—5 seconds)• personal preference
Likewise for experiment 2:

The criteria that human subjects use in judging singing skill
can also be looked into from the introspective comments.
Example features mentioned in the comments for experiment 1 include:••

tonal stability••

tonal stability
rhythmical stability•singing technique (e.g. vibrato, keeping a stable F0)•vocal expression and quality

Figure 3. Percentage of significant pairs.
(or at least, objectively taken) features such as tonal stability as judgment criteria, and that only a short sequence (3—5 sec.) is sufficient for judging good/poor. These points
give practical support for the realizability of such a scheme.

DISCUSSION
The results of I show that 391 pairs (88.9%) of subject
rankings were significant at the 5% level, and 260 pairs(65.0%) were significant at the 1% level. This suggests thatthe rankings are generally stable and in mutual agreement,
meaning that they are based more on common, objectivefeatures, contrary to the comments mentioning that evaluation is a matter of personal preference. The ρ values in Tables 3, 6, 8 all have positive (and in many cases, high)mean values, also indicating that the general tendency of
the rankings are stable. Furthermore, in the good/poor classification, none of the samples were completely divided
between good and poor ratings.

The results of II show that the subjects' rankings of F0
singing are stable in some cases (e.g. A—Japanese—male,A—Japanese—female, and B—English—female) but not soin others. High correlation rates are obtained when themelodies consist of relatively long notes, which require
higher singing skills. But together with the relatively low
overall values of the results of III, it can be said that F0
alone is not decisive for judging singing skills, and otheracoustic and musical features are incorporated in achieving
the high correlation rates in the results of I . One interesting point is that some comments for experiment 2 mentioned "vocal expression or quality", indicating that suchfeatures can (at least in a subjective sense) be recognizedeven with information of F0 alone.

Being such, the results of the labeling (good/poor) can betaken as a sufficiently reliable basis to be utilized in developing an automatic evaluation scheme. This is further supported by the fact that many comments refer to objective
ISBN 88-7395-155-4 © 2006 ICMPC1511

ICMPC9 Proceedings[4] Franco, H., Leonardo, N., Digalakis, V. & Ronen, O.
(2000). Combination of Machine Scores for Automatic
Grading of Pronunciation Quality. Speech Communication, 30, 121-130.

CONCLUSION
The results show that under the control of lyrics language,
singers' gender, and melody type (verse/chorus), the rankings given by the subjects are generally stable, indicatingthat they depend more on common, objective features rather
than reflecting subjective preference. This makes the results
reliable enough to be used as a referendum for developing
automatic singing evaluation schemes.

[5] Kendall, M. & Gibbons, J. D. (1990). Rank Correlation
Methods. New York: Oxford University Press.
[6] Goto, M. & Nishimura, T. (2005). AIST Humming Database: Music Database for Singing Research. The Special Interest Group Notes of IPSJ (MUS), 2005 (82), 712. (in Japanese)

Further experiments will be conducted in various other settings to explore singing skills in more detail. Work on identifying the key acoustic properties that underlie humanjudgments is also in progress.

[7] Goto, M., Hashiguchi, H., Nishimura, T. & Oka, R.
(2002). RWC Music Database: Popular, Classical, andJazz Music Databases. in Proceedings of the 3rd International Conference on Music Information Retrieval(ISMIR2002), 287-288.

ACKNOWLEDGMENTS
This paper utilized the RWC Music Database (Popular Music) and AIST Humming Database.

REFERENCES[8] Goto, M., Hashiguchi, H., Nishimura, T. & Oka, R.
(2003). RWC Music Database: Music Genre Databaseand Musical Instrument Sound Database. in Proceedingsof the 4th International Conference on Music
Information Retrieval (ISMIR2003), 229-230.

[1] Sundberg, J. (1987). The Science of the Singing Voice.
Illinois: the Northern Illinois University Press.
[2] Saitou, T., Unoki, M. & Akagi, M. (2005). Developmentof an F0 Control Model Based on F0 Dynamic
Characteristics for Singing-voice Synthesis. SpeechCommunication, 46, 405-417.

[9] Goto, M., Itou, K. & Hayamizu, S. (1999). A Real-timeFilled Pause Detection System for Spontaneous Speech
Recognition. in Proceedings of the 6th European Conference on Speech Communication and Technology (Eurospeech ‘99), 227-230.

[3] Omori, K., Kacker, A., Carroll, L. M., Riley, W. D. &Blaugrund, S. M. (1996). Singing Power Ratio: Quantiative Evaluation of Singing Voice Quality. Journal ofVoice, 10 (3), 228-235.

ISBN 88-7395-155-4 © 2006 ICMPC1512