International Journal of Semantic Computing
Vol. 10, No. 1 (2016) 27–52
c World Scienti¯c Publishing Company°
DOI: 10.1142/S1793351X1640002X

Musical Similarity and Commonness Estimation Based on
Probabilistic Generative Models of Musical Elements

Tomoyasu Nakano*,‡, Kazuyoshi Yoshii†,§ and Masataka Goto*,¶*National Institute of Advanced Industrial

Science and Technology (AIST)
Ibaraki 305-8568, Japan†
Kyoto University
Kyoto 606-8501, Japan‡t.nakano@aist.go.jp§yoshii@kuis.kyoto-u.ac.jp¶
m.goto@aist.go.jp

This paper proposes a novel concept we call musical commonness, which is the similarity of asong to a set of songs; in other words, its typicality. This commonness can be used to retrieve
representative songs from a set of songs (e.g. songs released in the 80s or 90s). Previous researchon musical similarity has compared two songs but has not evaluated the similarity of a song to aset of songs. The methods presented here for estimating the similarity and commonness ofpolyphonic musical audio signals are based on a uni¯ed framework of probabilistic generativemodeling of four musical elements (vocal timbre, musical timbre, rhythm, and chord progression). To estimate the commonness, we use a generative model trained from a song set instead of
estimating musical similarities of all possible song-pairs by using a model trained from eachsong. In experimental evaluation, we used two song-sets: 3278 Japanese popular music songs
and 415 English songs.Twenty estimated song-pair similarities for each element and each songset were compared with ratings by a musician. The comparison with the results of the expert
ratings suggests that the proposed methods can estimate musical similarity appropriately.
Estimated musical commonnesses are evaluated on basis of the Pearson product-moment correlation coe±cients between the estimated commonness of each song and the number of songs
having high similarity with the song. Results of commonness evaluation show that a song having
higher commonness is similar to songs of a song set.
Keywords: Musical similarity; musical commonness; typicality; latent Dirichlet allocation;
variational Pitman-Yor language model.

1. Introduction
The digitization of music and the distribution of content over the web have greatly
increased the number of musical pieces that listeners can access but are also causing
problems for both listeners and creators. Listeners ¯nd that selecting music is getting
more di±cult, and creators ¯nd that their creations can easily just disappear into

27

28

T. Nakano, K. Yoshii & M. Gotomusicalsimilarity
song asong alow
highmusicalcommonness
song bsong asong csong bsong set A

high

e.g.)
a particular genre
a personal collectiona specific playlist

lowFig. 1. Musical similarity and commonness.

obscurity. Musical similarity [1–3] between two songs can help with these problems
because it provides a basis for retrieving musical pieces that closely match a listener's
favorites, and several similarity-based music information retrieval (MIR) systems [1,
3–7] and music recommender systems [2, 8] have been proposed. None, however, has
focused on the musical similarity of a song to a set of songs, such as those in aparticular genre or personal collection, those on a speci¯c playlist, or those released ina given year or a decade.
This paper focuses on musical similarity and musical commonness that can be usedin MIR systems and music recommender systems. As shown in Fig. 1, we de¯ne musicalcommonness as a similarity assessed by comparing a song with a set of songs. The moresimilar a song is to songs in that set, the higher its musical commonness. Our de¯nitionis based on central tendency, which, in cognitive psychology, is one of the determinantsof typicality [9]. Musical commonness can be used to recommend a representative or
introductory song for a set of songs (e.g. songs released in the 80s), and it can help
listeners understand the relationship between a song and such a song set.
To estimate musical similarity and commonness, we propose a generativemodeling of four musical elements: vocal timbre, musical timbre, rhythm, and chord
progression (Fig. 2). Previous works on music information retrieval have extracted
various featuresa [1, 3, 5] including these four elements. We selected them to achieve
diverse similarities and commonnesses via our estimation method. Two songs are
considered to be similar if one has descriptions (e.g. chord names) that have a highprobability in a model of the other. This probabilistic approach has previously been
mentioned/used to compute similarity between two songs [10, 11]. To computecommonness for each element, a generative model is derived for a set of songs. A songis considered to be common to that set if the descriptions of the song have a highprobability in the derived model.
The following sections describe our approach and the experimental results of its
evaluation. Section 2 presents acoustic features and probabilistic generative models
and Sec. 3 describes estimation experiments and their evaluation. Section 4 considers our contribution in relation to previous works, Sec. 5 discusses the importanceof musical commonness, and Sec. 6 concludes the paper, with directions for futurework.

a For example, the following have all been used as features: singer voice, timbre, rhythm, onset, beat,tempo, melody, pitch, bass, harmony, tonality, chord, key, loudness, musical structure, and lyrics.

Musical Similarity and Commonness Estimation Based on Probabilistic Generative Models

29

2. Methods
From polyphonic musical audio signals including a singing voice and sounds ofvarious musical instruments we ¯rst extract vocal timbre, musical timbre, andrhythm and estimate chord progression. We then model the timbres and rhythm byusing a vector quantization method and latent Dirichlet allocation (LDA) [12]. The
chord progression is modeled by using a variable-order Markov process (up to atheoretically in¯nite order) called the variable-order Pitman-Yor language model(VPYLM) [13, 14].
When someone compares two pieces of music, they may feel that they share somefactors that characterize their timbres, rhythms and chord progressions, even if theycannot articulate exactly what these factors are. We call these ``latent factors" andwould like to estimate them from low-level features. This is di±cult to do for individual songs, but using the above methods (LDA and VPYLM) we can do so using
many songs.
Finally, for each element we calculate two probabilities (Fig. 2). One is for similarity estimation and is calculated by using a generative model trained from amusical piece (this model is called a song model). The other is for commonness
estimation and is calculated by using a generative model trained from a set of musicalpieces (this model is called a song-set model).

song amusicalelements

songmodelsmusicalelements

vocal timbre
musical timbre
rhythmchord progression

generative model

vocal timbre
musical timbre
rhythmchord progression

generative model
generative model
generative model

song bcalculated probability = musical similarity
song set A

musicalelements

song-setmodelsmusicalelements

vocal timbre
musical timbre
rhythmchord progression

generative model

vocal timbre
musical timbre
rhythmchord progression

generative model
generative model
generative model

song bcalculated probability = musical commonness

Fig. 2. Musical similarity and commonness based on probabilistic generative modeling of four musicalelements: vocal timbre, musical timbre, rhythm, and chord progression.

2.1. Similarity and commonness: Vocal timbre, musical timbre, andrhythm
The method used to train song models of vocal timbre, musical timbre, and rhythm isbased on a previous work [15] on modeling vocal timbre. In addition, we propose amethod to train song-set models under the LDA-based modeling.

30

T. Nakano, K. Yoshii & M. Goto

2.1.1. Extracting acoustic features: Vocal timbre
We use the mel-frequency cepstral coe±cients of the LPC spectrum of the vocal(LPMCCs) and the F0 of the vocal to represent vocal timbre because they aree®ective for identifying singers [11, 15]. In particular, the LPMCCs represent thecharacteristics of the singing voice well, since singer identi¯cation accuracy is greater
when using LPMCCs than when using the standard mel-frequency cepstral coe±cients (MFCCs) [11].
We ¯rst use Goto's PreFEst [16] to estimate the F0 of the predominant melody
from an audio signal and then the F0 is used to estimate the F0 and the LPMCCs of
the vocal. To estimate the LPMCCs, the vocal sound is re-synthesized by using asinusoidal model based on the estimated vocal F0 and the harmonic structure estimated from the audio signal. At each frame the F0 and the LPMCCs are combined
as a feature vector.
Then reliable frames (frames little in°uenced by accompaniment sound) are selected by using a vocal GMM and a non-vocal GMM (see [11] for details). Feature
vectors of only the reliable frames are used in the following processes (model training
and probability estimation).
2.1.2. Extracting acoustic features: Musical timbre
We use mel-frequency cepstral coe±cients (MFCCs) [17], their derivatives(MFCCs), and power to represent musical timbre, combining them as a feature
vector. This combined feature vector is often used in speech recognition. The MFCCs
are musical timbre features used in music information retrieval [18] and are robust toframe/hop sizes and lossy encoding provided that a minimum bitrate of approximately 160 Kbps is used [19].
2.1.3. Extracting acoustic features: Rhythm
To represent rhythm we use the °uctuation patterns (FPs) designed to describe therhythmic signature of musical audio [18, 20]. They are features e®ective for music
information retrieval [18] and for evaluating musical complexity with respect totempo [21].
We ¯rst calculate the speci¯c loudness sensation for each frequency band by using
an auditory model (i.e. the outer-ear model) and the Bark frequency scale. The FPs
are then obtained by using a FFT to calculate the amplitude modulation of theloudness sensation and weighting its coe±cients on the basis of a psychoacoustic
model of the °uctuation strength (see [18, 20] for details). Finally, the number of vector
dimensions of the FPs was reduced by using principle component analysis (PCA).
2.1.4. Quantization
All acoustic feature vectors of each element are converted to symbolic time series byusing a vector quantization method called the k-means algorithm. In that algorithm

Musical Similarity and Commonness Estimation Based on Probabilistic Generative Models

31

the vectors are normalized by subtracting the mean and dividing by the standard
deviation and then the normalized vectors are quantized by prototype vectors(centroids) trained previously. Hereafter, we call the quantized symbolic time series
acoustic words.
2.1.5. Probabilistic generative model~D g.
~ ¼ fX~1 ; . . . ; X
The observed data we consider for LDA are D independent songs X~d is Nd acoustic words X~d ¼ f~A song X
xd;1 ; . . . ; ~xd;Nd g. The size of the acoustic words
vocabulary is equivalent to the number of clusters of the k-means algorithm (¼ V ),~x d;n is a V -dimensional ``1-of-V " vector (a vector with one element containing a 1~dand all other elements containing a 0). The latent variable of the observed X~d ¼ f~is Z
zd;1 ; . . . ; ~zd;Nd g. The number of topics is K, so zd;n indicates a K-dimensional
1-of-K vector. Hereafter, all latent variables of D songs are indicated~ ¼ fZ~1 ; . . . ; Z~D g.
ZThe full joint distribution of the LDA model is given by~ ¼ pðX
~ Z
~
~; Z
~; 
~ jZ
~; Þpð
~j~
pðX
~; Þ
 Þpð~
 ÞpðÞ

ð1Þ

where ~
 indicates the mixing weights of the multiple topics (D of the K-dimensional~ indicates the unigram probability of each topic (K of the V -dimenvector) and 
sional vector). The ¯rst two terms are likelihood functions, and the other two areprior distributions. The likelihood functions are de¯ned as
!x
d;n;v
Nd Y
D
V
K
Y
Y
Y
z
~ ¼
~ jZ
~; Þ
pðX
 d;n;k
ð2Þ
k;v

d¼1

n¼1

v¼1

k¼1

and
~j~
pðZ
Þ ¼

Nd Y
D
V
Y
Y
d¼1

n¼1

z

d;n;k
 d;k
:

ð3Þ

v¼1

We then introduce conjugate priors as follows:
pð~
Þ ¼

D
Y

Dirð~
 d j~
 ð0Þ Þ ¼

d¼1

~ ¼
pðÞ

K
Y
k¼1

D
Y

C ð~
 ð0Þ Þ

d¼1
ð0Þ

~ Þ¼
~k j
Dirð

K
Y
k¼1

K
Y

 1
 d;k
;

ð0Þ

ð4Þ

ð0Þ

ð5Þ

k¼1
ð0Þ

~ Þ
C ð

V
Y

 1
 k;v
;

v¼1

~ð0Þ are~ are products of Dirichlet distributions, ~ð0Þ and 
where p(~ ) and p()hyperparameters of prior distributions (with no observation), and C ð~
 ð0Þ Þ andð0Þ~ Þ are normalization factors.
C ð

32

T. Nakano, K. Yoshii & M. Goto

2.1.6. Similarity estimation
The similarity between song a and song b is represented as a probability of song b
calculated using a song model of song a. This probability pg ðbjaÞ is de¯ned as follows:b
1 X~
log pð~x b;n jE½~
 a ; E½Þ;Nb n¼1Nlog pg ðbjaÞ ¼~ ¼
 a ; E½Þpð~
x b;n jE½~

K 
X


E½a;k   E½k;v  ;

ð6Þ
ð7Þ

k¼1

where E½ is the expectation of a Dirichlet distribution and v is the corresponding
index (the word id) of the K -dimensional 1-of-K observation vector ~x b;n .
2.1.7. Commonness estimation
To estimate the commonness, we propose a method for obtaining a generative modelfrom a song set without using the LDA-model-training process again. In this case,
hyperparameters d;k of the posterior distribution can be interpreted as e®ective numbers of observations of the corresponding values of the 1-of-K observation vector ~x d;n .
This means that a song-set model of a song set A can be obtained by summing~d ¼ fd;1 ; . . . ; d;K g. This model ~A is de¯ned as follows:
those hyperparameters X~ð0Þ þ ~ð0Þ ;~d  
~A ¼
ð8Þ


d2A

where the prior (~ ð0Þ ) is added just once. Musical commonness between the song set
A and the song a is represented as a probability of song a that is calculated using thesong-set model of the song set A: log pg ðajAÞ.
2.2. Similarity and commonness: Chord progression
We ¯rst estimate key and chord progression by using modules of Songle [22], a web
service for active music listening.
Before modeling, estimated results of chord progression are normalized. The rootnote is shifted so that the key will be /C/, °at notes ([) are uni¯ed into sharp notes(]), and the ¯ve variants of major chords with di®erent bass notes are uni¯ed (theyare dealt with as the same chord type). When same chord types continue, they arecollected into a single occurrence (e.g. /C C C/ into /C/).
2.2.1. Probabilistic generative model
For modeling of chord progression of a set of musical pieces, the VPYLM used asa song-set model is trained using a song set used to compute musical commonness.
In the song modeling process, however, suitable training cannot be done using only
a Bayesian model (VPYLM) because the amount of training data is not su±cient.

Musical Similarity and Commonness Estimation Based on Probabilistic Generative Models

33

To deal with this problem, we use as a song model a trigram model trained by
maximum likelihood estimation.
2.2.2. Similarity and commonness estimation
Similarity and commonness are represented by using as the generative probability
the inverse of the perplexity (average probability of each chord). To avoid the zerofrequency problem, chord similarity between two songs is estimated by calculating
weighted mean probabilities of the song model and the song-set model. The weights
are ð1  rÞ and r, respectively (with r set to 10 5 ).
3. Experiments
The proposed methods were tested in experiments evaluating the estimated similarity (Experiments A1 and A2) and the estimated commonness (Experiments B1
and B2).
3.1. Dataset
The song set used for model training, similarity estimation, and commonness estimation comprised 3278 Japanese popular songsb that appeared on a popular music
chart in Japan (http://www.oricon.co.jp/) and were placed in the top twenty on
weekly charts appearing between 2000 and 2008. Here we refer to this song set as the
JPOP music database (JPOP MDB). The twenty artists focused on for similarity
evaluation are listed in Table 1.
Another song set used for model training, similarity estimation, and commonness
estimation comprised 415 English songs performed by various types of artists (solosingers, male/female singers, bands, or groups). They were taken from commercialmusic CDs (Billboard Top Rock 'n 'Roll Hits 1968–1972, Billboard Top Hits 1975–
1989, and GRAMMY NOMINEES 1996–2005). Here we refer to this song set as the
English music database (ENG MDB). The twenty artists focused on for similarity
evaluation are listed in Table 2.
The song set used for GMM/k-means/PCA training to extract the acoustic features consisted of 100 popular songs from the RWC Music Database (RWC-MDB-P2001) [23]. These 80 songs in Japanese and 20 in English re°ect styles of the Japanese
popular songs (J-Pop) and Western popular songs in or before 2001. Here we refer tothis song set as the RWC MDB.
3.2. Experimental settings
Conditions and parameters of the methods described in Sec. 2 are described here indetail.
b Note that some are English songs in them.

34

T. Nakano, K. Yoshii & M. Goto
Table 1. Singers of the 463 songs used in the experiments A1 and B1.
ID

Artist name

A
B
C
D
E
F
G
H
I
J
K
L
M
N
O
P
Q
R
S
T
Total

Ayumi Hamasaki
B'z
Morning Musume
Mai Kuraki
Kumi Koda
BoA
EXILE
L'Arc-en-Ciel
Rina Aiuchi
w-inds.
SOPHIAMika Nakashima
CHEMISTRYGackt
GARNET CROW
TOKIO
Porno Gra±tti
Ken Hirai
Every Little Thing
GLAY

Gender of vocalist(s)(* more than one singer)

Number of songs

femalemale
female*femalefemalefemalemale*
male
femalemale*
male
femalemale*
male
female
male*
male
male
femalemale
11 male. 9 female

33
28
28
27
25
24
24
24
24
23
22
22
21
21
20
20
20
20
19
19
463

Table 2. Singers of the 62 songs used in experiments A2 and B2.
ID

Artist name

BO
ST
U2
BB
BL
BS
CA
DJ
EJ
EM
EC
KT
PS
RK
RM
SC
SS
TH
TT
AC
Total

Billy Ocean
Sting
U2
Backstreet Boys
Blondie
Britney Spears
Christina Aguiler
Daryl Hall & John Oates
Elton John
EminemEric Clapton
KC & The Sunshine Band
Pointer Sisters
R. Kelly
Richard Marx
Sheryl Crow
StarshipThree Dog Night
Tommy James & The Shondells
Ace Of Base

Gender of vocalist(s)(* more than one singer)

Number of songs

male
male
male
male*
femalefemalefemale
male*
male
male
male
male*
female*male
male
female
male*
male*
male
female*14 male. 6 female

4
4
4
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
2
62

Musical Similarity and Commonness Estimation Based on Probabilistic Generative Models

35

3.2.1. Extracting acoustic features
For vocal timbre features, we targeted monaural 16-kHz digital recordings andextracted F0 and 12th-order LPMCCs every 10 ms. The analysis frame length
was 32 ms. To estimate the features, the vocal sound was re-synthesized by using
a sinusoidal model with the frequency and amplitude of the lth overtone(l ¼ 1; . . . ; 20). The F0 was calculated every ¯ve frames (50 ms), the order of LPC
analysis was 25, the number of Mel-scaled ¯lter banks was 15.
The feature vectors were extracted from each song, using as reliable vocal frames
the top 15% of the feature frames. Using the 100 songs of the RWC MDB, a vocal
GMM and a non-vocal GMM were trained by variational Bayesian inference [24]. We
set the number of Gaussians to 32 and set the hyperparameter of a Dirichlet distribution over the mixing coe±cients to 1.0. The trained GMMs were models inwhich the number of Gaussians was reduced, to 12 for the vocal GMM and to 27 forthe non-vocal GMM.
For musical timbre features, we targeted monaural 16-kHz digital recordings andextracted power, 12th-order MFCCs, and 12th-order MFCCs every 10 ms. The features were calculated every ¯ve frames (50 ms), the pre-emphasis coe±cients for
was 0.97, the number of Mel-scaled ¯lter banks was 15, and the cepstral lifteringcoe±cient was 22. The feature vectors were extracted from 15% of the frames of eachsong and those frames were selected randomly.
For rhythm-based features, we targeted monaural 11.025-kHz digital recordings
and extracted FPs by using the Music Analysis (MA) toolbox for Matlab [18]. A
1200-dimension FP vector was estimated every 3 seconds and the analysis frame
length was 6 seconds. We then reduced the number of vector dimensions by using
PCA based on the cumulative contribution ratio ( 95%). A projection matrix
for PCA was computed by using the 100 songs of the RWC MDB. Finally, a78-dimensional projection matrix was obtained.
The conditions described above (e.g. the 16- and 11.025-kHz sampling frequencies) were based on previous work [15, 18].
3.2.2. Quantization
To quantize the vocal features, we set the number of clusters of the k-means algorithm to 100 and used the 100 songs of the RWC MDB to train the centroids. This k
is same number used in our previous work [15]. The number of clusters used toquantize the musical timbre and rhythm features was set to 64 in this evaluation.
3.2.3. Chord estimation
With Songle, chords are transcribed using 14 chord types: major, major 6th, major
7th, dominant 7th, minor, minor 7th, half-diminished, diminished, augmented, and¯ve variants of major chords with di®erent bass notes (/2, /3, /5, /b7, and /7). The
resulting 168 chords (14 types  12 root notes) and one ``no chord" label are estimated (see [22] for details).

36

T. Nakano, K. Yoshii & M. Goto

3.2.4. Training the generative models
Training song models and song-set models of the 4 musical elements by LDA andVPYLM, we used all of the 3278 original recordings of the JPOP MDB and all of the
415 recordings of the ENG MDB.
The number of topics K was set to 100, and the model parameters of LDA weretrained using the collapsed Gibbs sampler [25]. The hyperparameters of the Dirichlet
distributions for topics and words were initially set to 1 and 0:1, respectively. The
conditions were based on our previous work [15].
The number of chords used to model chord progression was 97: the 8 chord types(major, major 6th, major 7th, dominant 7th, minor, minor 7th, diminished, augmented) for each of the 12 di®erent root notes, and one ``no chord" label(97 ¼ 8  12 þ 1).
3.2.5. Baseline methods
The baseline methods used to estimate similarity and commonness were simple
methods.
The baseline methods used to estimate the similarity of vocal timbre, musicaltimbre, and rhythm calculated the Euclidean distance between mean feature vectorsof two songs. In the baseline methods used to estimate the commonness of theseelements, the mean feature vectors were calculated for a song-set and used to calculate the Euclidean distance from a target song. Each mean vector was normalizedby subtracting the mean and dividing by the standard deviation.
To model chord progression, we used as a song model a unigram model trained by
maximum likelihood estimation. The baseline modeling of chord progression of a setof musical pieces, used as a song-set model the HPYLM n-gram model [26] (with n setto 1). To avoid the zero-frequency problem, chord similarity between two songs was
also estimated by calculating weighted mean probabilities of the song model and thesong-set model. The weights were ð1  rÞ and r, respectively (with r set to 10 5 ).
3.3. Experiment A1: Similarity estimation (JPOP MDB)
To evaluate musical similarity estimation based on probabilistic generative models,
experiment A1 used all 3278 songs for modeling and estimated the similarities of the
463 songs by the artists listed in Table 1 (DA1 ¼ 463). Those 463 songs were sung by
the twenty artists with the greatest number of songs in the modeling set. The
evaluation set was very diverse: artists include solo singers and bands, and a balanceof male and female vocalists.
3.3.1. Similarity matrix
We ¯rst estimated the similarities between the 463 songs with respect to the fourmusical elements. Figures 3(a) through (d) show the similarity matrix for each ofthese elements, and Fig. 4 shows the baseline results. In each ¯gure the horizontal

450

song No. (target of computing similarity)

song No. (target of computing similarity)

Musical Similarity and Commonness Estimation Based on Probabilistic Generative Models

400
350
300
250
200
150
100
50

100
200
300
400
artist ID
A B C D E F G H I J K L MN O P QR S T

450
400
350
300
250
200
150
100
50

100
200
300
400
artist ID
A B C D E F G H I J K L MN O P QR S T(b) musical timbresong No. (target of computing similarity)

song No. (target of computing similarity)(a) vocal timbre

450
400
350
300
250
200
150
100
50

100
200
300
400
artist ID
A B C D E F G H I J K L MN O P QR S T(c) rhythm

37

450
400
350
300
250
200
150
100
50

100
200
300
400
artist ID
A B C D E F G H I J K L MN O P QR S T(d) chord progression

Fig. 3. Similarities among all 463 songs by the artists listed in Table 1 (similarities estimated using
probabilistic generative models).

axis indicates song number (song model used as a query) and the vertical axis
indicates target song number for similarity computation.
A similarity matrix represents 214,369 (463  463) pairs, and in each of thematrices only the 46 target songs (10% of DA1 ) having the highest similarities foreach of the queries are colored black.
3.3.2. Comparing estimated similarities with expert human ratings
We next evaluated the song models by using expert ratings. Twenty song pairs
belonged to two groups, referred to as the top10 and bottom10. The top10 group

T. Nakano, K. Yoshii & M. Goto

450

song No. (target of computing similarity)

song No. (target of computing similarity)

38

400
350
300
250
200
150
100
50

100
200
300
400
artist ID
A B C D E F G H I J K L MN O P QR S T

450
400
350
300
250
200
150
100
50

100
200
300
400
artist ID
A B C D E F G H I J K L MN O P QR S T

450(b) musical timbresong No. (target of computing similarity)

song No. (target of computing similarity)(a) vocal timbre

400
350
300
250
200
150
100
50

100
200
300
400
artist ID
A B C D E F G H I J K L MN O P QR S T(c) rhythm

450
400
350
300
250
200
150
100
50

100
200
300
400
artist ID
A B C D E F G H I J K L MN O P QR S T(d) chord progression

Fig. 4. Baseline similarities among all 463 songs by the artists listed in Table 1.

included the ten song pairs having the highest similarities for each of the musicalelements, under the selection restriction that there was no overlapping of singer
names in the group. This means that this group comprised only pairs of songs sungby di®erent singers. The bottom10 group included the ten song pairs (also selectedunder the no-overlapping-name condition) having the lowest similarities for each of
the musical elements. Table 3 shows the top10 and bottom10 groups based on thesimilarity estimated using the proposed methods and the baseline methods.
A music expert (a male musician) who was professionally-trained for music at his
graduating school and had experience with audio mixing/mastering, writing lyrics,
and arrangement/composition of Japanese popular songs was asked to rate song-pairMusical Similarity and Commonness Estimation Based on Probabilistic Generative Models

39

Table 3. The twenty song pairs belonged to two groups: Experiment A1 (JPOP MDB).
Proposed method

Baseline method

Vocal
timbreMusicaltimbreRhythmChordprogression

Vocal
timbreMusicaltimbreRhythmChordprogression

L–O
F–S
J–I
A–D
B–Q
M–R
H–P
E–C
G–N
K–T

B–A
H–T
Q–K
M–R
D–L
S–I
P–N
O–G
E–F
J–C

K–G
I–S
D–C
E–Q
A–F
O–H
R–L
N–T
P–B
J–M

D–I
O–B
K–N
A–J
H–T
P–C
S–L
F–E
Q–G
M–R

top10
F–E
A–C
L–I
N–K
R–G
T–Q
B–H
D–O
M–J
P–S

F–E
M–I
P–J
B–S
K–D
A–O
H–N
T–C
G–R
L–Q

N–L
O–E
B–I
R–G
C–A
T–K
S–P
F–D
M–H
Q–J

F–E
M–I
P–J
B–S
K–D
A–O
H–N
T–C
G–R
L–Q

F–E
T–J
H–D
P–A
Q–L
O–B
G–S
C–N
M–K
R–I

G–J
O–E
C–B
T–R
Q–A
P–F
I–M
S–N
H–L
K–D

P–O
H–C
G–S
N–E
M–Q
B–R
K–F
L–D
T–J
A–I

bottom10
O–P
C–T
T–R
H–S
B–M
K–B
Q–N
P–F
J–A
E–M
D–K
N–D
S–G
A–J
H–F
L–Q
I–C
G–I
L–E
O–R

P–O
S–T
Q–B
H–D
I–G
M–K
A–L
N–C
R–E
F–J

N–T
L–H
O–B
E–A
S–P
C–F
J–K
R–M
G–I
D–Q

P–O
S–T
Q–B
H–D
I–G
M–K
A–L
N–C
R–E
F–J

(L – O, for example, means a song of singer L and a song of singer O)

similarity on a 7-point scale ranging from 1 (not similar) to 7 (very similar). Rating
to a precision of one decimal place (e.g. 1.5) was allowed.
Figure 5 shows the results of the rating by the musician, and Fig. 6 shows theresults of rating based on the baseline results. The statistics of the ratings are shownby box plots indicating median values, 1/4 quantiles, 3/4 quantiles, minimum values,
and maximum values. Testing the results by using Welch's t-test [27] revealed thatthe di®erences between the two groups were signi¯cant at the 0.1% level for vocal
and musical timbre, the 1% level for rhythm, and the 5% level for chord progression(Fig. 5).
3.4. Experiment A2: Similarity estimation (ENG MDB)
To evaluate musical similarity estimation based on probabilistic generative models,
experiment A2 used all 415 songs for modeling and estimated the similarities of thesongs by the artists listed in Table 2 (DA2 ¼ 62). Those 62 songs were sung by thetwenty artists with the greatest number of songs in the modeling set.
We evaluated the song models by using expert ratings. As in experiment A1,
twenty song pairs belonged to two groups, referred to as the top10 and bottom10.
Table 4 shows the top10 and bottom10 groups based on the similarity estimated7

p < 0.1%

41

similarity
ratingT. Nakano, K. Yoshii & M. Goto

similarity
rating40

71

p < 1%

4
1(b) musical timbre

similarity
ratingsimilarity
rating(a) vocal timbre7

p < 0.1%

4(c) rhythm7
p < 5%

41(d) chord progression7

p < 1%

41

similarity
ratingsimilarity
ratingFig. 5. Box plots showing the statistics for the song-pair similarity ratings by a musician: Experiment A1(JPOP MDB).

741(a) vocal timbre

p < 10%4
1(c) rhythm

7
similarity
ratingsimilarity
rating7(b) musical timbre

p < 0.1%

41(d) chord progression

Fig. 6. Box plots showing the statistics for the baseline song-pair similarity ratings: Experiment A1(JPOP MDB, baseline).

using the proposed methods and the baseline methods. Figure 7 shows the results of
the rating by the musician, and Fig. 8 shows the results of rating based on thebaseline results.
3.5. Discussion for experiments A1 and A2
From the similarity matrices for the JPOP MDB one sees that songs by the same
artist have high similarity for vocal timbre and musical timbre. For rhythm andchord progression, on the other hand, some songs by the same artist have highsimilarity (indicated by arrows in Figs. 3(c) and 3(d)) but most do not. These results
re°ect musical characteristics qualitatively and can be understood intuitively.
Although the similarity matrices for the ENG MDB are not shown, they indicateda similar tendency.

Musical Similarity and Commonness Estimation Based on Probabilistic Generative Models

41

Table 4. The twenty song pairs belonged to two groups: Experiment A2 (ENG MDB).
Proposed method

Baseline method

Vocal
timbreMusicaltimbreRhythmChordprogression

Vocal
timbreMusicaltimbreRhythmChordprogression

EC – RKBL – THEM – EJ
BB – TTKT – DJPS – BSCA – ACU2 – ST
SS – SCBO – RM

BB – RM
ST – EJ
U2 – TTSS – DJSC – RKBO – THBS – CAPS – BLEC – EM
AC – KT

RM – ST
BO – BBKT – U2
EJ – CADJ – EC
AC – BSTH – PSSC – RKSS – TTEM – BL

top10
PS – BLBB – SCU2 – TH
EJ – SSEC – SCBO – KTAC – SSEC – RKBB – KT
BL – DJRM – BO
PS – U2
DJ – TT
AC – CACA – EJ
BS – TT
RK – ST
RM – STEM – BSEM – TH

EJ – U2
DJ – KTBO – THEC – SSBL – CARK – SC
PS – ST
RM – TT
BS – EM
AC – BB

RM – ST
EC – KTCA – RKSS – U2
EJ – THDJ – TTBB – SCBL – PSAC – BSBO – EMEC – BLSC – THPS – TT
AC – SS
EJ – KT
RM – BO
BS – CAST – BBEM – U2
RK – DJBS – BBU2 – SSSC – KT
TH – CAEM – BO
TT – EC
RM – AC
EJ – PSDJ – ST
RK – BL

EJ – BB
DJ – SC
TH – KT
RK – BO
ST – PSU2 – BS
BL – EC
TT – EM
RM – CASS – AC

RM – TT
TH – SCRK – BLSS – PS
EJ – BO
DJ – U2
EC – CABB – BSKT – ACST – EMbottom10
SC – THSS – BSRM – U2
PS – BBCA – BBSC – BO
BL – BO
KT – EJ
EC – ACU2 – CAST – PSEM – ACSS – BSTH – EC
DJ – EMST – BLKT – RKRK – DJEJ – TT
TT – RM

SS – SCRM – EJ
TT – BB
RK – PSKT – EMEC – BO
TH – BLST – BSU2 – DJCA – AC

RM – RKTT – EJ
SS – SCU2 – STTH – BLKT – CAEC – BO
PS – BB
DJ – BSEM – AC

RM – PS
TH – BOSC – EC
DJ – BSU2 – BB
BL – CASS – ST
AC – EM
EJ – TTKT – RKEC – RK, for example, means a song of singer EC (Eric Clapton) and a song of singer RK (R. Kelly)7

p < 5%

41

similarity
ratingsimilarity
ratingOn the similarity matrix for rhythm, horizontal lines can be seen. This means that
there are songs that in most cases get high similarity regardless of which song is thequery song. On the other hand, there are also songs that get low similarity with mostquery songs. LDA topic distributions for both kinds are shown in Fig. 9. The former
7
4
1

top10

top10

7

(b) musical timbre
similarity
ratingsimilarity
rating(a) vocal timbre

41

7
p < 5%

41

top10(c) rhythm(d) chord progression

Fig. 7. Box plots showing the statistics for the song-pair similarity ratings by a musician: Experiment A2(ENG MDB).

T. Nakano, K. Yoshii & M. Goto

similarity
rating7

similarity
rating

42

4
1

7
4
1

top10

(a) vocal timbre(b) musical timbre
similarity
ratingsimilarity
rating

7
4
1

7
4
1

top10

(c) rhythm(d) chord progression

Fig. 8. Box plots showing the statistics for the baseline song-pair similarity ratings: Experiment A2(ENG MDB).

topic distribution (rhythm)
mixing weight0.2

a song of artist T

0.1
0a song of artist D

0.2

0.1
0
10

20

30

40

50

60

70

80

90 topic

Fig. 9. Top: topic distribution of a song that gets high similarity with most songs. Bottom: topic distribution of a song that gets low similarity with most songs.

kind's is °at and has some topics having value, and the latter kind's has a few topics
having value. On the similarity matrix for chord progression, there are query songs
that get high similarity with all other songs (e.g. a song of singer A) and there arequery songs that get low similarity with all other songs (see, e.g. Fig. 10: Top). In thebaseline unigram setting, on the other hand, the query song of singer A has di®erent
similarities with all other songs (Fig. 10: Bottom).
The comparison with the results of the expert ratings suggests that the proposed
methods can estimate musical similarity appropriately. The musician was asked forthe judgment (evaluation) criteria after the all ratings, and they were as follows:
vocal timbre (1) ringing based on the distribution of the harmonic overtone,(2) pitch (F0 , fundamental frequency), (3) degree of breathy voice.
musical timbre (1) composition of the musical instruments, (2) balance of loudnessof each instruments, reverberation, and dynamics via the audio mixing/mastering.
(3) music genre,rhythm (1) rhythm pattern, (2) beat structure or degree of shu®le (swing),(3) music genre, (4) tempo.

similarity(chord progression)

0
-2
-4
-6
-8
-10
-12

similarity(chord progression)

Musical Similarity and Commonness Estimation Based on Probabilistic Generative Models

0

proposed method
query (singer A)

query (singer F)

baseline method
query (singer A)

query (singer F)

43

-5
-10
-15
0

50

100

150

200

250

300

350

400

song No.

Fig. 10. Blue: similarity between a query song and others. Red: similarity between a query song andothers with low similarity in most cases.

chord progression (1) the pattern of chord progression, (2) chords used in thesongs.
To improve the performance with regard to all elements, conditions such as those forextracting acoustic features, for quantization, for chord estimation, and for modeltraining can be considered in future work. Especially, in the results of the comparisonof the estimated similarity with the expert ratings for the ENG MDB, there is no
signi¯cant di®erence for musical timbre and rhythm (Fig. 7). The musician said that
there are di®erences for the musical timbre and rhythm among the songs because of
the released date of songs are wide-ranging (1968–2005).
3.6. Experiment B1: Commonness estimation (JPOP MDB)
To evaluate musical commonness estimation based on probabilistic generativemodels, experiment B1 also used the 3278 songs of the JPOP MDB to train the songset models and for evaluating each musical element.
When evaluating the commonness estimation method, we ¯rst evaluated thenumber of songs having high similarity. For example, in Fig. 1 the song a has many
similar songs in the song set A. If a song having higher (lower) commonness is very
similar (is not similar) to songs of a song set.
Figure 11 shows the relationships between the estimated commonness of songs
contained in the JPOP MDB to the number of songs having high similarity. We usedas the threshold for deciding the similarity of an element to be high the 3/4 quantile
value of all similarities among all 10,745,284 (3278  3278) possible song-pairs in the
JPOP MDB.
The Pearson product-moment correlation coe±cients are shown in each part ofthe ¯gure and are also listed in Table 5. The reliability of the estimated similarity canbe evaluated by using the results shown in Figs. 5 and 6. The asterisk mark (*) and44

T. Nakano, K. Yoshii & M. Goto
The number of songs

similarity

110

1861

-3

-4.23
-7
1000

2000

3000

1000

(b) musical timbre

the number of songs having high similarity(a) vocal timbre
2000(* 0.766)

4000

3000

1861

110
0
-5

(* 0.834)

0
-3.4

-4.5

-3.7

(d) chord progression(c) rhythm

4000(* 0.735)

0song ID2000

3/4
quantile
value(* 0.677)

0
-4
0
-2
commonness
commonness(*x): Pearson product-moment correlation coefficient-7

Fig. 11. Relationships between estimated commonness of the four elements of each song and the numberof songs having high similarity with the song: Experiment B1 (JPOP MDB).

Table 5. Pearson product-moment correlation coef¯cients between estimated commonness of the four
elements each song and the number of songs having
high similarity with the song: Experiment B1 (JPOPMDB). Conditions: S) The number of songs having
high similarity, SB) The number of songs having highsimilarity (baseline), C) Commonness, CB) Commonness (baseline).
Correlation coe±cients
Element

Condition

C

CB

vocal
musical timbre
rhythmchord progression

S**
S**
S*
S

0.766
0.834
0.735
0.670

0.175
0.350
0.650
0.886

vocal
musical timbre
rhythmchord progression

SB*
SB
SB
SB**

0.137
0.402
0.774
0.759

0.960
0.958
0.898
0.846

Estimated similarity is comparable to ratings by amusician** at the 0.1% signi¯cance level (Figs. 5 and 6)* at the 1% signi¯cance level (Figs. 5 and 6)

Musical Similarity and Commonness Estimation Based on Probabilistic Generative Models

45

the double-asterisk mark (**) in Table 5 indicate di®erences between the top10 andbottom10 groups that are signi¯cant at the 1% and 0.1% levels, respectively.
Under conditions of the relatively reliable similarities (``vocal S**", ``musicaltimbre S**", and ``rhythm S*") the correlation coe±cient of the proposed method(``C": 0.766, 0.834, and 0.735) are larger than those of the baseline method (``CB":
0.175, 0.350, and 0.650). The results suggest that the more similar a song is to songsof the song set, the higher its musical commonness in the proposed method. Although
two coe±cients of the condition ``vocal SB*" and ``chord progression SB**" arepositive values (``C": 0.137 and 0.759), the corresponding coe±cients for the baseline
method (``CB": 0.960 and 0.846) are larger. The improvement of the correlation
coe±cients is a subject for future investigation.
3.7. Experiment B2: Commonness estimation (ENG MDB)
To evaluate musical commonness estimation based on probabilistic generativemodels, experiment B2 also used the 415 songs of the ENG MDB to train the song-set
models and for evaluating each musical element. As in experiment B1, we evaluatedthe number of songs having high similarity.
Figure 12 shows the relationships between the estimated commonness of songs
contained in the ENG MDB to the number of songs having high similarity. We usedas the threshold for deciding the similarity of an element to be high the 3/4 quantile
value of all similarities among all 172,223 (415  415) possible song-pairs in the ENG
MDB.
The Pearson product-moment correlation coe±cients are shown in each part ofthe ¯gure and are also listed in Table 6. The reliability of the estimated similarity canbe evaluated by using the results shown in Figs. 7 and 8. The asterisks (*) in Table 6(b) musical timbre

the number of songs having high similarity(a) vocal timbre
400

400(* 0.888)

0
-4.5
400

0
-3

(c) rhythm(* 0.761)0-8(* 0.857)

-4.3

-3.8

(d) chord progression
400(* 0.740)0-300commonness
commonness(*x): Pearson product-moment correlation coefficientFig. 12. Relationships between estimated commonness of the four elements each song and the number ofsongs having high similarity with the song: Experiment B2 (ENG MDB).

46

T. Nakano, K. Yoshii & M. Goto
Table 6. Pearson product-moment correlation coe±cients
between estimated commonness of the four elements of eachsong and the number of songs having high similarity withthe song: Experiment B2 (ENG MDB). Conditions: S) The
number of songs having high similarity, SB) The number ofsongs having high similarity (baseline), C) Commonness, CB)
Commonness (baseline).
Correlation coe±cients
Element

Condition

C

CB

vocal
musical timbre
rhythmchord progression

S*
S
S
S*

0.888
0.857
0.761
0.740

0.0500.421
0.8200.707

vocal
musical timbre
rhythmchord progression

SB
SB
SB
SB

0.172
0.104
0.004
0.891

0.959
0.077
0.070
0.908

*Estimated similarity is comparable to ratings by a musician
at the 5% signi¯cance level (Figs. 7 and 8).

indicate di®erences between the top10 and bottom10 groups that are signi¯cant at
the 5% level.
Under conditions of the relatively reliable similarities (``vocal S*" and ``chord
progression S*") the values of the correlation coe±cient of the proposed method(``C": 0.888 and 0.740) are bigger than the baseline method (``CB": 0.050 and
0.707). Moreover, the coe±cients based on the proposed methods (row ``S" andcolumn ``C") are all high (greater than 0.7). The results suggest that the similarity
and commonness estimated using the proposed methods have a mutual relationship.
This relationship is useful to use the commonness under the typicality de¯nition. In
other words, the commonness can be used instead of the number of songs having highsimilarity among the songs.
3.8. Application of commonness in terms of vocal timbre
Only the song-set models of vocal timbre can be evaluated quantitatively by using
the singer's gender. These models are integrated song models with di®erent ratios of
the number of male singers to female singers.
To train song-set models, we used 14 songs by di®erent solo singers (6 male and 8
female) from the JPOP MDB. We trained three types of song-set models: one trainedby using all 14 songs, one trained by using one female song and all 6 male songs, andone is trained by using one male song and all 8 male songs.
Figure 13 shows the vocal timbre commonnesses based on the 3 di®erent song-set
models. When a model with a high proportion of female songs is used, the commonness of songs sung by females is higher than the commonness of songs sung by
males (and vice versa). In Fig. 14 the statistics of the commonnesses are shown by
box plots. The results suggest the commonnesses can re°ect vocal tract features.

Musical Similarity and Commonness Estimation Based on Probabilistic Generative Models

female vocalists

male vocalists

7 songs

3 song-set models:commonness

0.03

1

2

3

4

5

6

47

14 songs
9 songs 1 7 8 9 10 11 12 13 14
ALL
B, A, D, E, F, I, L, O, S

7

B, H, N, Q, R, T, A

proposed method

00.4

baseline method

0

1

2

3

4

5

6

7

8

9

10

11

12

13

B

H

N

Q

R

T

A

D

E

F

I

L

O

male vocalists

14 song No.
S artist ID

female vocalists

Fig. 13. Vocal timbre commonness based on 3 di®erent song-set models for 14 songs (6 male and female).

male vocalists

0.03

7 songs

Model 1

1

2

3

4

5

6

Model 2Model 2

9 songs 1 7 8 9 10 11 12 13 14
B, A, D, E, F, I, L, O, S

Model 1Model 2

0.4

0.02
0.01
0

female vocalists

7

B, H, N, Q, R, T, A

baselinecommonness

Model 1

p < 1%

p < 1%

male female(1-6) (7-14)

male female(1-6) (7-14)

p < 1%

0male female(1-6) (7-14)

male female(1-6) (7-14)

Fig. 14. Box plots showing the statistics for the vocal timbre commonness (Fig. 13).

3.9. Applying the proposed method to other elements
The proposed LDA-based method and the VPYLM-base method can be applied tovarious music-related characteristics. Lyrics, for example, are an important elementof music in various genres, especially in popular music. Since the LDA and the
VPYLM were originally proposed for text analysis, they can be used for lyrics
modeling. In fact, there are three papers on work that used lyrics for LDA-based
music retrieval [28–30].
Figure 15 shows that the results of the expert rating comparing with LDA-based
estimated similarities and correlation coe±cients between estimated commonness of
the lyrics each song and the number of songs having high similarity with the song.
This results are based on a set of lyrics of 1996 songs: 1896 Japanese popular lyrics apart of the JPOP MDB and 100 lyrics of the RWC MDB. 340 lyrics of the twenty
artists with the greatest number of lyrics in the lyrics set are used to select the twenty

lyrics
7

p < 1%

41
top10

bottom10

the number of songs
having high similarity

T. Nakano, K. Yoshii & M. Goto
similarity rating

48

2000

lyrics(* 0.761)

0

-10
-5
commonness(*x): Pearson product-moment correlation coefficientFig. 15. Relationships between estimated commonness of the lyrics each song and the number of songs
having high similarity with the song.

lyrics pairs, the top10 and bottom10. The number of topics K was set to 100, and
MeCab [31] was used for the morphological analysis of Japanese lyrics. 19,390 words(morphemes) is the vocabulary size in the 1996 lyrics. The results suggest that the
proposed method can be applied to music lyrics.
As other characteristics, artist properties provide di®erent type of relations
among songs. The artist-level information obtained from web such as Wikipedia(http://www.wikipedia.org/) and its commonness (typicality) can be used tovisualize the relations [32].
4. Related Studies
Musical similarity is a central concept of MIR and is also important for purposes
other than retrieval. For example, the use of similarity to automatically classify
musical pieces (into genres, music styles, etc.) is being researched [1, 2], and musicalsimilarity can also be used for music auto-tagging [3]. However, each of theseapplications is di®erent from the idea of musical commonness: musical similarity is
usually de¯ned by comparing two songs, music classi¯cation is de¯ned by classifying
a given song into one out of a set of categories (category models, centroids, etc.), andmusic auto-tagging is de¯ned by comparing a given song to a set of tags (tag models,the closest neighbors, etc.). To the best of our knowledge, there is no research about
the automatic estimation of musical commonness, de¯ned as the typicality of a songwith respect to a set of songs. Therefore, we think that music commonness is a novel
concept which can be used to retrieve representative songs from a set of songs.
This paper has proposed a uni¯ed framework of probabilistic generative modelingto estimate musical similarity and commonness. To realize the framework, we haveintroduced latent analysis of music. There are previous works related to latent
analysis of music, such as music retrieval based on LDA of lyrics and melodic features[28], lyrics retrieval based on LDA [29], assessing quality of lyrics topic model (LDA)[30], chord estimation based on LDA [33, 34], combining document and music spaces
by latent semantic analysis [35], music recommendation by social tag and latentsemantic analysis [36], and music similarity based on the hierarchical Dirichlet process [37]. In contrast to these previous reports, we showed that LDA and VPYLM canbe combined to do musical similarity and commonness estimation using four musicalelements (vocal timbre, musical timbre, rhythm, and chord progression) and lyrics.

Musical Similarity and Commonness Estimation Based on Probabilistic Generative Models

49

5. Discussion
The contributions of this paper are 1) proposing the concept of musical commonness,2) showing that a generative model trained from a song set can be used for commonness estimation (instead of estimating musical similarities of all possible songpairs by using a model trained from each song), 3) showing how to evaluate theestimated commonness.
Described as in Sec. 1, the amount of digital content that can be accessed bypeople has been increasing and will continue to do so in the future. This is desirable
but unfortunately makes it easier for the work of content creators to become buried
within a huge amount of content, making it harder for viewers and listeners to selectcontent. Furthermore, since the amount of similar content is also increasing, creators
will be more concerned that their content might invite unwarranted suspicion ofplagiarism. All kinds of works are in°uenced by existing content, and it is di±cult
to avoid the unconscious creation of content partly similar in some way to priorcontent.
However, human ability with regard to similarity is limited. Judging similarity
between two songs one hears is a relatively simple task but takes time. One simply
does not have enough time to search a million songs for similar content. Moreover,while humans are able to make accurate judgments based on past experience, theirability to judge ``commonness" or ``typicality"  the probability of an event's
occurrence  is limited. When an uncommon event happens to be frequently observed recently, for example, people tend to wrongly assume that it is likely to occur.
And when a frequent event happens to not be encountered, people tend to wrongly
assume that it is rare. Consequently, with the coming of an ``age of billions ofcreators" in which anyone can enjoy creating and sharing works, the monotonic
increase in content means that there is a growing risk that one's work will be denounced as being similar to someone else's. This could make it di±cult for people tofreely create and share content.
The musical commonness proposed in this paper can help create an environment
in which specialists and general users alike can know the answers to the questions``What is similar here?" and ``How often does this occur?" Here we aim to make itpossible for people to continue creating and sharing songs without worry. Furthermore, we want to make it easy for anyone to enjoy the music content creation
process, and we want to do this by developing music-creation support technology
enabling ``high commonness" elements (such as chord progressions and conventional
genre-dependent practices) to be used as knowledge common to mankind. We also
want to promote a proactive approach to encountering and appreciating content by
developing music-appreciation support technology that enables people to encounternew content in ways based on its similarity to other content.
We hope to contribute to the creation of a culture that can mutually coexist withpast content while paying appropriate respect to it. This will become possible bysupporting a new music culture that enables creators to take delight in ¯nding their

50

T. Nakano, K. Yoshii & M. Goto

content being reused, in much the same way that researchers take delight in ¯nding
their articles being cited. We feel that the value of content cannot be measured by
the extent to which it is not similar to other content and that pursuing originality at
all costs does not necessarily bring joy to people. Fundamentally, content has value
by inducing an emotional and joyous response in people. We would like to make it amatter of common sense that content with emotional appeal and high-quality form
has value. In fact, we would like to see conditions in which it is exactly the referring
to many works that gives content its value, similar to the situation with academic
papers. Through this approach, we aim to create a content culture that emphasizes
emotionally touching experiences.

6. Conclusions and Future Work
This paper describes an approach to musical similarity and commonness estimationthat is based on probabilistic generative models: LDA and the VPYLM. Four musical elements are modeled: vocal timbre, musical timbre, rhythm, and chord progression. The commonness can be estimated by using song-set models, which is easierthan estimating the musical similarities of all possible pairs of songs.
The experimental results showed that our methods are appropriate for estimate
musical similarity and commonness. And these methods are potentially applicablewith other elements of music, such as lyrics. The probability calculation can beapplied not only to a musical piece but also to a part of a musical piece. This meansthat musical commonness is also useful to creators because a musical element thathas high commonness (e.g. a chord progression) is an established expression and canbe used by anyone creating and publishing musical content.
This paper showed the e®ectiveness of the proposed methods with song sets ofdi®erent sizes. The JPOP MDB was used as a large song set (more than 1000 songs)
and the ENG MDB was used as a medium-size song set (between 100 and 1000
songs). In [32] we used musical commonness for visualization and changing playbackorder with a small song set (less than 100 songs) as a personal music playlist. And theexperimental results in Sec. 3.8 also show the e®ectiveness of the musical commonness with a small song set.
Since this paper focused on the above four elements, we plan to use melody (e.g.
F0 ) as the next step. Future work will also include the integration ofs generative
probabilities based on di®erent models, calculating probabilities of parts of one song,
investigating e®ective features, and developing an interface for music listening or
creation by leveraging musical similarity and commonness.

Acknowledgments
This paper utilized the RWC Music Database (Popular Music). This work was
supported in part by CREST, JST.

Musical Similarity and Commonness Estimation Based on Probabilistic Generative Models

51

References[1] M. Goto and K. Hirata, Recent studies on music information processing, Acoustical
Science and Technology 25(6) (2004) 419–425.
[2] P. Knees and M. Schedl, A survey of music similarity and recommendation from music
context data, ACM Trans. on Multimedia Computing, Communications and Applications 10(1) (2013) 1–21.
[3] M. Schedl, E. Gomez and J. Urbano, Music information retrieval: Recent developments
and applications, Foundations and Trends in Information Retrieval 8(2–3) (2014) 127–261.
[4] B. Pardo, Ed., Special issue: Music information retrieval, Commun. ACM 49(8) (2006)
28–58.
[5] M. Casey, R. Veltkamp, M. Goto, M. Leman, C. Rhodes and M. Slaney, Content-based
music information retrieval: Current directions and future challenges, in Proc. IEEE 96(4) (2008) 668–696.
[6] J. S. Downie, The music information retrieval evaluation exchange (2005–2007): A
window into music information retrieval research, Acoust. Sci. & Tech. 29 (2008) 247–
255.
[7] J. S. Downie, D. Byrd and T. Crawford, Ten years of ISMIR: Re°ections on challenges
and opportunities, in Proc. 10th Int. Society for Music Inf. Retrieval Conf., 2009, pp. 13–
18.
[8] Y. Song, S. Dixon and M. Pearce, A survey of music recommendation systems and future
perspectives, in Proc. 9th Int. Symp. on Computer Music Modeling and Retrieval, 2012,
pp. 395–410.
[9] L. W. Barsalou, Ideals, central tendency, and frequency of instantiation as determinantsof graded structure in categories, Journal of Experimental Psychology: Learning,
Memory, and Cognition 11(4) (1985) 629–654.
[10] J.-J. Aucouturier and F. Pachet, Music similarity measures: What's the use? in Proc. 3rd
Int. Conf. Music Inf. Retrieval, 2002, pp. 157–163.
[11] H. Fujihara, M. Goto, T. Kitahara and H. G. Okuno, A modeling of singing voice robust
to accompaniment sounds and its application to singer identi¯cation and vocal-timbresimilarity based music information retrieval, IEEE Trans. on Audio, Speech, and Language Processing 18(3) (2010) 638–648.
[12] D. M. Blei, A. Y. Ng and M. I. Jordan, Latent Dirichlet allocation, Journal of Machine
Learning Research 3 (2003) 993–1022.
[13] D. Mochihashi and E. Sumita, The in¯nite Markov model, in Proc. Advances in Neural
Information Processing Systems 20, 2007, pp. 1017–1024.
[14] K. Yoshii and M. Goto, A vocabulary-free in¯nity-gram model for nonparametricbayesian chord progression analysis, in Proc. 12th Int. Society for Music Inf. Retrieval
Conf., 2014, pp. 645–650.
[15] T. Nakano, K. Yoshii and M. Goto, Vocal timbre analysis using latent Dirichlet allocation and cross-gender vocal timbre similarity, in Proc. 39th Int. Conf. on Acoustics,
Speech and Signal Processing, 2014, pp. 5239–5343.
[16] M. Goto, A real-time music scene description system: Predominant-F0 estimation for
detecting melody and bass lines in real-world audio signals, Speech Communication
43(4) (2004) 311–329.
[17] B. Logan, Mel frequency cepstral coe±cients for music modeling, in Proc. Int. Symp.
Music Inf. Retrieval, 2000, pp. 1–11.
[18] E. Pampalk, Computational models of music similarity and their application to music
information retrieval, Ph.D. dissertation, Vienna Inst. of Tech., 2006.

52

T. Nakano, K. Yoshii & M. Goto[19] J. Urbano, D. Bogdanov, P. Herrera, E. Gomez and X. Serra, What is the e®ect of audio
quality on the robustness of MFCCs and chroma features? in Proc. 15th Int. Society forMusic Inf. Retrieval Conf., 2014, pp. 573–578.
[20] E. Pampalk, A. Rauber and D. Merkl, Contentbased organization and visualization ofmusic archives, in Proc. ACM Multimedia, 2002, pp. 570–579.
[21] M. Mauch and M. Levy, Structural change on multiple time scales as a correlate ofmusical complexity, in Proc. ISMIR, 2011, pp. 489–494.
[22] M. Goto, K. Yoshii, H. Fujihara, M. Mauch and T. Nakano, Songle: A web service foractive music listening improved by user contributions, in Proc. 12th Int. Society forMusic Inf. Retrieval Conf., 2011, pp. 311–316.
[23] M. Goto, H. Hashiguchi, T. Nishimura and R. Oka, RWC music database: Popular,classical, and jazz music databases, in Proc. 3rd Int. Conf. Music Inf. Retrieval, 2002,
pp. 287–288.
[24] C. M. Bishop, Pattern Recognition and Machine Learning (Springer-Verlag, New York,
2006).
[25] T. L. Gri±ths and M. Steyvers, Finding scienti¯c topics, in Proc. Natl. Acad. Sci. USA,
Vol. 1 (2004), pp. 5228–5235.
[26] Y. W. Teh, A hierarchical bayesian language model based on Pitman-Yor processes, in
Proc. Joint Conf. of the Int. Committee on Computational Linguistics and the Association for Computational Linguistics, 2006, pp. 985–992.
[27] B. L. Welch, The signi¯cance of the di®erence between two means when the population
variances are unequal, Biometrika 29 (1938) 350–362.
[28] E. Brochu and N. de Freitas, ``name that song!": A probabilistic approach to querying on
music and text, in Proc. of Advances in Neural Information Processing Systems, 2002,
pp. 1505–1512.
[29] S. Sasaki, K. Yoshii, T. Nakano, M. Goto and S. Morishima, LyricsRadar: A lyrics
retrieval system based on latent topics of lyrics, in Proc. 15th Int. Society for Music Inf.
Retrieval Conf., 2014, pp. 585–590.
[30] L. Sterckx, T. Demeester, J. Deleu, L. Mertens and C. Develder, Assessing quality ofunsupervised topics in song lyrics, in Advances in Information Retrieval, 2014, pp. 547–
552.
[31] T. Kudo, MeCab: Yet another part-of-speech and morphological analyzer, http://mecab.
sourceforge.net/.
[32] T. Nakano, J. Kato, M. Hamasaki and M. Goto, PlaylistPlayer: An interface using
multiple criteria to change the playback order of a music playlist, in Proc. 21st ACM Int.
Conf. on Intelligent User Interfaces, 2016, pp. 186–190.
[33] D. J. Hu and L. K. Saul, A probabilistic topic model for unsupervised learning of musical
key-pro¯les, in Proc. of 10th Int. Society for Music Inf. Retrieval Conf., 2009, pp. 441–
446.
[34] D. J. Hu and L. K. Saul, A probabilistic topic model for music analysis, in Proc. of 29th
Annual Conference on Neural Information Processing Systems, 2009, pp. 1–4.
[35] R. Takahashi, Y. Ohishi, N. Kitaoka and K. Takeda, Building and combining document
and music spaces for music query-by-webpage system, in Proc. of 9th Annual Conf. of theInt. Speech Communication Association, 2008, pp. 2020–2023.
[36] P. Symeonidis, M. M. Ruxanda, A. Nanopoulos and Y. Manolopoulos, Ternary semantic
analysis of social tags for personalized music recommendation, in Proc. of 9th Int. Conf.
on Music Inf. Retrieval, 2008, pp. 219–224.
[37] M. Ho®man, D. Blei and P. Cook, Content-based musical similarity computation using
the hierarchical dirichlet process, in Proc. of 9th Int. Conf. on Music Inf. Retrieval, 2008,
pp. 349–354.