2015 IEEE International Symposium on Multimedia

0XVLFDO 6LPLODULW\ DQG &RPPRQQHVV (VWLPDWLRQEDVHG RQ 3UREDELOLVWLF *HQHUDWLYH 0RGHOV
7RPR\DVX 1DNDQR.D]X\RVKL <RVKLL

0DVDWDND *RWRNational Institute of Advanced Industrial
Science and Technology (AIST), Japant.nakano@aist.go.jp

Kyoto University,Japanyoshii@kuis.kyoto-u.ac.jp

National Institute of Advanced Industrial
Science and Technology (AIST), Japanm.goto@aist.go.jp

Abstract²7KLV SDSHU SURSRVHV D QRYHO FRQFHSW ZH FDOO
musical commonness ZKLFK LV WKH VLPLODULW\ RI D VRQJ WR D
VHW RI VRQJV LQ RWKHU ZRUGV LWV typicality 7KLV FRPPRQQHVV
FDQ EH XVHG WR UHWULHYH UHSUHVHQWDWLYH VRQJV IURP D VRQJ VHW
e.g. VRQJV UHOHDVHG LQ WKH V RU V  3UHYLRXV UHVHDUFKRQ PXVLFDO VLPLODULW\ KDV FRPSDUHG WZR VRQJV EXW KDV QRWHYDOXDWHG WKH VLPLODULW\ RI D VRQJ WR D VHW RI VRQJV 7KH
PHWKRGV SUHVHQWHG KHUH IRU HVWLPDWLQJ WKH VLPLODULW\ DQG
FRPPRQQHVV RI SRO\SKRQLF PXVLFDO DXGLR VLJQDOV DUH EDVHG
RQ D XQL¿HG IUDPHZRUN RI SUREDELOLVWLF JHQHUDWLYH PRGHOLQJ RIIRXU PXVLFDO HOHPHQWV YRFDO WLPEUH PXVLFDO WLPEUH UK\WKPDQG FKRUG SURJUHVVLRQ  7R HVWLPDWH WKH FRPPRQQHVV ZH XVH D
JHQHUDWLYH PRGHO WUDLQHG IURP D VRQJ VHW LQVWHDG RI HVWLPDWLQJPXVLFDO VLPLODULWLHV RI DOO SRVVLEOH VRQJSDLUV E\ XVLQJ D PRGHOWUDLQHG IURP HDFK VRQJ ,Q H[SHULPHQWDO HYDOXDWLRQ ZH XVHG
 SRSXODU PXVLF VRQJV (VWLPDWHG VRQJSDLU VLPLODULWLHV DUHFRPSDUDEOH WR UDWLQJV E\ D PXVLFLDQ DW WKH  VLJQL¿FDQFHOHYHO IRU YRFDO DQG PXVLFDO WLPEUH DW WKH  OHYHO IRU UK\WKP
DQG WKH  OHYHO IRU FKRUG SURJUHVVLRQ 5HVXOWV RI FRPPRQQHVVHYDOXDWLRQ VKRZ WKDW WKH KLJKHU WKH PXVLFDO FRPPRQQHVV LV WKH
PRUH VLPLODU D VRQJ LV WR VRQJV RI D VRQJ VHWsong asong asong bsong asong bﬁ

ﬁ

lowmusicalcommonness

high

ﬁ

ﬁ

low
musicalcommonness

ﬁ)LJXUH 

low
high

ﬁ

song bsong set Aﬁ ﬁ ﬁ ﬁ ﬁ
ﬁ ﬁ ﬁ ﬁ ﬁsong set Bﬁ ﬁ ﬁ ﬁ ﬁ
ﬁ ﬁ ﬁ ﬁ ﬁ

0XVLFDO VLPLODULW\ DQG FRPPRQQHVVD VRQJ ZLWK D VHW RI VRQJV 7KH PRUH VLPLODU D VRQJ LVWR VRQJV LQ WKDW VHW WKH KLJKHU LWV PXVLFDO FRPPRQQHVV
2XU GH¿QLWLRQ LV EDVHG RQ central tendency ZKLFK LQ
FRJQLWLYH SV\FKRORJ\ LV RQH RI WKH GHWHUPLQDQWV RI typicality>@ 0XVLFDO FRPPRQQHVV FDQ EH XVHG WR UHFRPPHQG D
UHSUHVHQWDWLYH RU LQWURGXFWRU\ VRQJ IRU D VRQJ VHW DQG LW FDQKHOS OLVWHQHUV XQGHUVWDQG WKH UHODWLRQVKLS EHWZHHQ D VRQJ
DQG D VRQJ VHW7R HVWLPDWH PXVLFDO VLPLODULW\ DQG FRPPRQQHVV ZH
SURSRVH D JHQHUDWLYH PRGHOLQJ RI IRXU PXVLFDO HOHPHQWVYRFDO WLPEUH PXVLFDO WLPEUH UK\WKP DQG FKRUG SURJUHVVLRQ)LJXUH   7ZR VRQJV DUH FRQVLGHUHG WR EH VLPLODU LI RQH KDV
GHVFULSWLRQV e.g. FKRUG QDPHV WKDW KDYH D KLJK SUREDELOLW\
LQ D PRGHO RI WKH RWKHU 7KLV SUREDELOLVWLF DSSURDFK KDV
SUHYLRXVO\ EHHQ XVHG WR FRPSXWH VLPLODULW\ EHWZHHQ WZRVRQJV >@ >@ 7R FRPSXWH FRPPRQQHVV IRU HDFK HOHPHQWD JHQHUDWLYH PRGHO LV GHULYHG IRU D VHW RI VRQJV $ VRQJ LVFRQVLGHUHG WR EH FRPPRQ WR WKDW VHW LI WKH GHVFULSWLRQV RIWKH VRQJ KDYH D KLJK SUREDELOLW\ LQ WKH GHULYHG PRGHOKeywordsPXVLFDO VLPLODULW\ PXVLFDO FRPPRQQHVV W\SLFDO
LW\ ODWHQW 'LULFKOHW DOORFDWLRQ YDULDWLRQDO 3LWPDQ<RU ODQJXDJHPRGHO

, , 1752'8&7,21
7KH GLJLWL]DWLRQ RI PXVLF DQG WKH GLVWULEXWLRQ RI FRQWHQW
RYHU WKH ZHE KDYH JUHDWO\ LQFUHDVHG WKH QXPEHU RI PXVLFDO
SLHFHV WKDW OLVWHQHUV FDQ DFFHVV EXW DUH DOVR FDXVLQJ SUREOHPVIRU ERWK OLVWHQHUV DQG FUHDWRUV /LVWHQHUV ¿QG WKDW VHOHFWLQJPXVLF LV JHWWLQJ PRUH GLI¿FXOW DQG FUHDWRUV ¿QG WKDW WKHLU
FUHDWLRQV FDQ HDVLO\ MXVW GLVDSSHDU LQWR REVFXULW\ 0XVLFDO
VLPLODULW\ >@ >@ EHWZHHQ WZR VRQJV FDQ KHOS ZLWK WKHVH
SUREOHPV EHFDXVH LW SURYLGHV D EDVLV IRU UHWULHYLQJ PXVLFDO
SLHFHV WKDW FORVHO\ PDWFK D OLVWHQHU¶V IDYRULWHV DQG VHYHUDO
VLPLODULW\EDVHG PXVLF LQIRUPDWLRQ UHWULHYDO 0,5 V\VWHPV>@±>@ DQG PXVLF UHFRPPHQGHU V\VWHPV >@ >@ KDYH EHHQ
SURSRVHG 1RQH KRZHYHU KDV IRFXVHG RQ WKH PXVLFDO VLPLODULW\ RI D VRQJ WR D VHW RI VRQJV VXFK DV WKRVH LQ D SDUWLFXODU
JHQUH RU SHUVRQDO FROOHFWLRQ WKRVH RQ D VSHFL¿F SOD\OLVW RU
WKRVH UHOHDVHG LQ D JLYHQ \HDU RU D GHFDGH
7KLV SDSHU IRFXVHV RQ PXVLFDO VLPLODULW\ DQG musicalcommonness WKDW FDQ EH XVHG LQ 0,5 V\VWHPV DQG PXVLFUHFRPPHQGHU V\VWHPV $V VKRZQ LQ )LJXUH  ZH GH¿QH
PXVLFDO FRPPRQQHVV DV D VLPLODULW\ DVVHVVHG E\ FRPSDULQJ
978-1-5090-0379-2/15 $31.00 © 2015 IEEE
DOI 10.1109/ISM.2015.102

musicalsimilarity,, 0 (7+2'6)URP SRO\SKRQLF PXVLFDO DXGLR VLJQDOV LQFOXGLQJ D
VLQJLQJ YRLFH DQG VRXQGV RI YDULRXV PXVLFDO LQVWUXPHQWV ZH¿UVW H[WUDFW YRFDO WLPEUH PXVLFDO WLPEUH DQG UK\WKP DQG
197

song amusicalelements

songmodelsmusicalelements

song b

vocal timbre
musical timbre
rhythmchord progression

generative modelﬁ

vocal timbre
musical timbre
rhythmchord progressionﬁsong set A

musicalelements

song-setmodelsmusicalelements

ﬁ ﬁ
ﬁ ﬁ
ﬁ ﬁ

vocal timbre
musical timbre
rhythmchord progression

generative model

vocal timbre
musical timbre
rhythmchord progression

generative model
generative model
generative modelXVLQJ D VLQXVRLGDO PRGHO EDVHG RQ WKH HVWLPDWHG YRFDO F0
DQG WKH KDUPRQLF VWUXFWXUH HVWLPDWHG IURP WKH DXGLR VLJQDO$W HDFK IUDPH WKH ΔF0 DQG WKH /30&&V DUH FRPELQHG DV
D IHDWXUH YHFWRU
7KHQ reliable frames IUDPHV OLWWOH LQÀXHQFHG E\ DFFRPSDQLPHQW VRXQG DUH VHOHFWHG E\ XVLQJ D YRFDO *00 DQG
D QRQYRFDO *00 VHH >@ IRU GHWDLOV  )HDWXUH YHFWRUV RIRQO\ WKH UHOLDEOH IUDPHV DUH XVHG LQ WKH IROORZLQJ SURFHVVHVPRGHO WUDLQLQJ DQG SUREDELOLW\ HVWLPDWLRQ 2) Extracting acoustic features: Musical timbre: :H XVH
PHOIUHTXHQF\ FHSVWUDO FRHI¿FLHQWV 0)&&V  WKHLU GHULYDWLYHV Δ0)&&V  DQG ΔSRZHU WR UHSUHVHQW PXVLFDO WLPEUHFRPELQLQJ WKHP DV D IHDWXUH YHFWRU 7KH 0)&&V DUH PXVLFDOWLPEUH IHDWXUHV XVHG LQ PXVLF LQIRUPDWLRQ UHWULHYDO >@ DQG
WKLV IHDWXUH YHFWRU LV RIWHQ XVHG LQ VSHHFK UHFRJQLWLRQ
3) Extracting acoustic features: Rhythm: 7R UHSUHVHQW
UK\WKP ZH XVH WKH ÀXFWXDWLRQ SDWWHUQV )3V GHVLJQHG WR
GHVFULEH WKH UK\WKPLF VLJQDWXUH RI PXVLFDO DXGLR >@ >@
7KH\ DUH IHDWXUHV HIIHFWLYH IRU PXVLF LQIRUPDWLRQ UHWULHYDO>@ DQG IRU HYDOXDWLQJ PXVLFDO FRPSOH[LW\ ZLWK UHVSHFW WR
WHPSR >@:H ¿UVW FDOFXODWH WKH specic loudness sensation IRU HDFKIUHTXHQF\ EDQG E\ XVLQJ DQ DXGLWRU\ PRGHO i.e. WKH RXWHUHDU PRGHO DQG WKH %DUN IUHTXHQF\ VFDOH 7KH )3V DUH
WKHQ REWDLQHG E\ XVLQJ D ))7 WR FDOFXODWH WKH DPSOLWXGHPRGXODWLRQ RI WKH ORXGQHVV VHQVDWLRQ DQG ZHLJKWLQJ LWV
FRHI¿FLHQWV RQ WKH EDVLV RI D SV\FKRDFRXVWLF PRGHO RI WKH
uctuation strength VHH >@ >@ IRU GHWDLOV  )LQDOO\ WKH
QXPEHU RI YHFWRU GLPHQVLRQV RI WKH )3V ZDV UHGXFHG E\
XVLQJ SULQFLSOH FRPSRQHQW DQDO\VLV 3&$ 4) Quantization: $OO DFRXVWLF IHDWXUH YHFWRUV RI HDFKHOHPHQW DUH FRQYHUWHG WR V\PEROLF WLPH VHULHV E\ XVLQJ DYHFWRU TXDQWL]DWLRQ PHWKRG FDOOHG WKH kPHDQV DOJRULWKP,Q WKDW DOJRULWKP WKH YHFWRUV DUH QRUPDOL]HG E\ VXEWUDFWLQJ
WKH PHDQ DQG GLYLGLQJ E\ WKH VWDQGDUG GHYLDWLRQ DQG WKHQ
WKH QRUPDOL]HG YHFWRUV DUH TXDQWL]HG E\ SURWRW\SH YHFWRUVFHQWURLGV WUDLQHG SUHYLRXVO\ +HUHDIWHU ZH FDOO WKH TXDQWL]HG V\PEROLF WLPH VHULHV acoustic words
5) Probabilistic generative model: 7KH REVHUYHG GDWD =ZH FRQVLGHU IRU /'$ DUH D LQGHSHQGHQW VRQJV X{X1 , ..., XD } $ VRQJ Xd LV Nd DFRXVWLF ZRUGV Xd ={xd,1 , ..., xd,Nd } 7KH VL]H RI WKH DFRXVWLF ZRUGV YRFDEXODU\LV HTXLYDOHQW WR WKH QXPEHU RI FOXVWHUV RI WKH kPHDQVDOJRULWKP = V  xd,n LV D V GLPHQVLRQDO ³RIV ´ YHFWRU
D YHFWRU ZLWK RQH HOHPHQW FRQWDLQLQJ D 1 DQG DOO RWKHU
HOHPHQWV FRQWDLQLQJ D 0  7KH ODWHQW YDULDEOH RI WKH REVHUYHG
 d LV Z
 d = {zd,1 , ..., zd,N } 7KH QXPEHU RI WRSLFV LV K VR
X
dzd,n LQGLFDWHV D KGLPHQVLRQDO RIK YHFWRU +HUHDIWHU DOO
 D }
 = {Z 1 , ..., Z
ODWHQW YDULDEOHV RI D VRQJV DUH LQGLFDWHG Z
7KH IXOO MRLQW GLVWULEXWLRQ RI WKH /'$ PRGHO LV JLYHQ E\

calculated probability = musical similarity

generative model
generative model
generative model

song bﬁ

calculated probability = musical commonness)LJXUH 
0XVLFDO VLPLODULW\ DQG FRPPRQQHVV EDVHG RQ SUREDELOLVWLF
JHQHUDWLYH PRGHOLQJ RI IRXU PXVLFDO HOHPHQWV YRFDO WLPEUH PXVLFDO WLPEUHUK\WKP DQG FKRUG SURJUHVVLRQHVWLPDWH FKRUG SURJUHVVLRQ :H WKHQ PRGHO WKH WLPEUHV DQG
UK\WKP E\ XVLQJ D YHFWRU TXDQWL]DWLRQ PHWKRG DQG ODWHQW'LULFKOHW DOORFDWLRQ /'$ >@ 7KH FKRUG SURJUHVVLRQ LVPRGHOHG E\ XVLQJ D YDULDEOHRUGHU 0DUNRY SURFHVV XS WR DWKHRUHWLFDOO\ LQ¿QLWH RUGHU FDOOHG WKH YDULDEOHRUGHU 3LWPDQ<RU ODQJXDJH PRGHO 93</0 >@ >@:KHQ VRPHRQH FRPSDUHV WZR SLHFHV RI PXVLF WKH\ PD\
IHHO WKDW WKH\ VKDUH VRPH IDFWRUV WKDW FKDUDFWHUL]H WKHLU
WLPEUHV UK\WKPV DQG FKRUG SURJUHVVLRQV HYHQ LI WKH\ FDQQRW
DUWLFXODWH H[DFWO\ ZKDW WKHVH IDFWRUV DUH :H FDOO WKHVH ³latent
factors´ DQG ZRXOG OLNH WR HVWLPDWH WKHP IURP ORZOHYHOIHDWXUHV 7KLV LV GLI¿FXOW WR GR IRU LQGLYLGXDO VRQJV EXW XVLQJWKH DERYH PHWKRGV /'$ DQG 93</0 ZH FDQ GR VR XVLQJPDQ\ VRQJV)LQDOO\ IRU HDFK HOHPHQW ZH FDOFXODWH WZR SUREDELOLWLHV)LJXUH   2QH LV IRU VLPLODULW\ HVWLPDWLRQ DQG LV FDOFXODWHG
E\ XVLQJ D JHQHUDWLYH PRGHO WUDLQHG IURP D PXVLFDO SLHFHWKLV PRGHO LV FDOOHG D song model  7KH RWKHU LV IRU FRPPRQQHVV HVWLPDWLRQ DQG LV FDOFXODWHG E\ XVLQJ D JHQHUDWLYHPRGHO WUDLQHG IURP D VHW RI PXVLFDO SLHFHV WKLV PRGHO LV
FDOOHG D song-set model 
A. Similarity and commonness: Vocal timbre, musical timbre, and rhythm
7KH PHWKRG XVHG WR WUDLQ VRQJ PRGHOV RI YRFDO WLPEUHPXVLFDO WLPEUH DQG UK\WKP LV EDVHG RQ D SUHYLRXV ZRUN >@RQ PRGHOLQJ YRFDO WLPEUH ,Q DGGLWLRQ ZH SURSRVH D PHWKRG
WR WUDLQ VRQJVHW PRGHOV XQGHU WKH /'$EDVHG PRGHOLQJ
1) Extracting acoustic features: Vocal timbre: :H XVH WKH
PHOIUHTXHQF\ FHSVWUDO FRHI¿FLHQWV RI WKH /3& VSHFWUXP RIWKH YRFDO /30&&V DQG WKH ΔF0 RI WKH YRFDO WR UHSUH
VHQW YRFDO WLPEUH EHFDXVH WKH\ DUH HIIHFWLYH IRU LGHQWLI\LQJ
VLQJHUV >@ >@:H ¿UVW XVH *RWR¶V 3UH)(VW >@ WR HVWLPDWH WKH F0 RI WKH
SUHGRPLQDQW PHORG\ IURP DQ DXGLR VLJQDO DQG WKHQ WKH F0 LV
XVHG WR HVWLPDWH WKH ΔF0 DQG WKH /30&&V RI WKH YRFDO 7R
HVWLPDWH WKH /30&&V WKH YRFDO VRXQG LV UHV\QWKHVL]HG E\

 = p(X|
 Z|

 Z,
 π , φ) Z, φ)p( π )p(π )p(φ)
p(X,ZKHUH π LQGLFDWHV WKH PL[LQJ ZHLJKWV RI WKH PXOWLSOH WRS LQGLFDWHV WKHLFV D RI WKH KGLPHQVLRQDO YHFWRU DQG φ198

XQLJUDP SUREDELOLW\ RI HDFK WRSLF K RI WKH V GLPHQVLRQDOYHFWRU  7KH ¿UVW WZR WHUPV DUH OLNHOLKRRG IXQFWLRQV DQG WKH
RWKHU WZR DUH SULRU GLVWULEXWLRQV 7KH OLNHOLKRRG IXQFWLRQV DUH
GH¿QHG DV
K
xd,n,v
Nd 
D 
V

 z
d,n,k



p(X|Z, φ) =φ


B. Similarity and commonness: Chord progression:H ¿UVW HVWLPDWH NH\ DQG FKRUG SURJUHVVLRQ E\ XVLQJPRGXOHV RI 6RQJOH >@ D ZHE VHUYLFH IRU DFWLYH PXVLF
OLVWHQLQJ%HIRUH PRGHOLQJ HVWLPDWHG UHVXOWV RI FKRUG SURJUHVVLRQDUH QRUPDOL]HG 7KH URRW QRWH LV VKLIWHG VR WKDW WKH NH\ ZLOO
EH & ÀDW QRWHV  DUH XQL¿HG LQWR VKDUS QRWHV   DQG WKH¿YH YDULDQWV RI PDMRU FKRUGV ZLWK GLIIHUHQW EDVV QRWHV DUHXQL¿HG WKH\ DUH GHDOW ZLWK DV WKH VDPH FKRUG W\SH  :KHQ
VDPH FKRUG W\SHV FRQWLQXH WKH\ DUH FROOHFWHG LQWR D VLQJOH
RFFXUUHQFH e.g. & & & LQWR & 1) Probabilistic generative model: )RU PRGHOLQJ RI FKRUG
SURJUHVVLRQ RI D VHW RI PXVLFDO SLHFHV WKH 93</0 XVHG DV
D VRQJVHW PRGHO LV WUDLQHG XVLQJ D VRQJ VHW XVHG WR FRPSXWH
PXVLFDO FRPPRQQHVV ,Q WKH VRQJ PRGHOLQJ SURFHVV KRZHYHU VXLWDEOH WUDLQLQJ FDQQRW EH GRQH XVLQJ RQO\ D %D\HVLDQPRGHO 93</0 EHFDXVH WKH DPRXQW RI WUDLQLQJ GDWD LV QRWVXI¿FLHQW 7R GHDO ZLWK WKLV SUREOHP ZH XVH DV D VRQJ PRGHOD WULJUDP PRGHO WUDLQHG E\ PD[LPXP OLNHOLKRRG HVWLPDWLRQ
2) Similarity and commonness estimation: 6LPLODULW\ DQG
FRPPRQQHVV DUH UHSUHVHQWHG E\ XVLQJ DV WKH JHQHUDWLYH
SUREDELOLW\ WKH LQYHUVH RI WKH perplexity DYHUDJH SUREDELOLW\RI HDFK FKRUG  7R DYRLG WKH ]HURIUHTXHQF\ SUREOHP FKRUGVLPLODULW\ EHWZHHQ WZR VRQJV LV HVWLPDWHG E\ FDOFXODWLQJZHLJKWHG PHDQ SUREDELOLWLHV RI WKH VRQJ PRGHO DQG WKH VRQJVHW PRGHO 7KH ZHLJKWV DUH (1 − r) DQG r UHVSHFWLYHO\ r LV
VHW WR 10−5 k,v

d=1 n=1 v=1

DQG

k=1

Nd 
D 
V


 π) =
p(Z|

z

d,n,k
πd,k
.



d=1 n=1 v=1:H WKHQ LQWURGXFH FRQMXJDWH SULRUV DV IROORZV
p(π ) =D'LU(πd |
α(0) ) =

d=1

 =
p(φ)

K


D


C(
α(0) )d=1

 k |β (0) ) =
'LU(φ

k=1

K

k=1

K


(0)

α −1
πd,k
,



k=1

 (0) )
C(β

V


(0)

φβk,v −1 ,v=1 DUH SURGXFWV RI 'LULFKOHW GLVWULEXWLRQVZKHUH S π DQG S φα (0) DQG β (0) DUH K\SHUSDUDPHWHUV RI SULRU GLVWULEXWLRQV
 (0) ) DUH QRUZLWK QR REVHUYDWLRQ  DQG C(α(0) ) DQG C(βPDOL]DWLRQ IDFWRUV
6) Similarity estimation: 7KH VLPLODULW\ EHWZHHQ VRQJa DQG VRQJ b LV UHSUHVHQWHG DV D SUREDELOLW\ RI VRQJ b
FDOFXODWHG XVLQJ D VRQJ PRGHO RI VRQJ a 7KLV SUREDELOLW\
pg (b|a) LV GH¿QHG DV IROORZVb1 log p(xb,n | E[πa ], E[φ]),Nb n=1Nlog pg (b|a) = =p(xb,n | E[πa ], E[φ])K(E[πa,k ] · E[φk,v ]) ,,,, ( ;3(5,0(176

7KH SURSRVHG PHWKRGV ZHUH WHVWHG LQ H[SHULPHQWV HYDOXDWLQJ WKH HVWLPDWHG VLPLODULW\ ([SHULPHQW $ DQG WKH
HVWLPDWHG FRPPRQQHVV ([SHULPHQW % 

k=1

A. Dataset

ZKHUH E[·] LV WKH H[SHFWDWLRQ RI D 'LULFKOHW GLVWULEXWLRQ
DQG v LV WKH FRUUHVSRQGLQJ LQGH[ WKH ZRUG LG RI WKH K
GLPHQVLRQDO RIK REVHUYDWLRQ YHFWRU xb,n 7) Commonness estimation: 7R HVWLPDWH WKH FRPPRQ
QHVV ZH SURSRVH D PHWKRG IRU REWDLQLQJ D JHQHUDWLYH PRGHOIURP D VRQJ VHW ZLWKRXW XVLQJ WKH /'$PRGHOWUDLQLQJ
SURFHVV DJDLQ ,Q WKLV FDVH K\SHUSDUDPHWHUV αd,k RI WKH
SRVWHULRU GLVWULEXWLRQ FDQ EH LQWHUSUHWHG DV HIIHFWLYH QXPEHUVRI REVHUYDWLRQV RI WKH FRUUHVSRQGLQJ YDOXHV RI WKH RIK
REVHUYDWLRQ YHFWRU xd,n 
7KLV PHDQV WKDW D VRQJVHW PRGHO RI D VRQJ VHW A
FDQ EH REWDLQHG E\ VXPPLQJ WKRVH K\SHUSDUDPHWHUV αd ={αd,1 , ..., αd,K } 7KLV PRGHO α
 A LV GH¿QHG DV IROORZVαd − α (0) ,
 (0) + ααA =7KH VRQJ VHW XVHG IRU WKH PRGHO WUDLQLQJ VLPLODULW\
HVWLPDWLRQ DQG FRPPRQQHVV HVWLPDWLRQ FRPSULVHG -DSDQHVH SRSXODU VRQJV WKDW DSSHDUHG RQ D SRSXODU PXVLF
FKDUW LQ -DSDQ KWWSZZZRULFRQFRMS DQG ZHUH SODFHG LQ
WKH WRS WZHQW\ RQ ZHHNO\ FKDUWV DSSHDULQJ EHWZHHQ DQG  +HUH ZH UHIHU WR WKLV VRQJ VHW DV WKH -323 PXVLF
GDWDEDVH 0'%  7KH WZHQW\ DUWLVWV IRFXVHG RQ IRU VLPLODULW\HYDOXDWLRQ DUH OLVWHG LQ 7DEOH ,
,Q DGGLWLRQ IRU *00kPHDQV3&$ WUDLQLQJ WR H[WUDFWWKH DFRXVWLF IHDWXUHV  SRSXODU VRQJV IURP WKH 5:&
0XVLF 'DWDEDVH 5:&0'%3 >@ ZHUH DOVR XVHG
7KHVH  VRQJ LQ -DSDQHVH DQG  LQ (QJOLVK UHÀHFW VW\OHVRI WKH -DSDQHVH SRSXODU VRQJV -3RS DQG :HVWHUQ SRSXODUVRQJV LQ RU EHIRUH  +HUH ZH UHIHU WKLV VRQJ VHW DV WKH
5:& 0'%

d∈A

ZKHUH WKH SULRU α (0) LV DGGHG MXVW RQFH 0XVLFDO FRPPRQQHVV EHWZHHQ WKH VRQJ VHW A DQG WKH VRQJ a LV UHSUHVHQWHG DV
D SUREDELOLW\ RI VRQJ a WKDW LV FDOFXODWHG XVLQJ WKH VRQJVHWPRGHO RI WKH VRQJ VHW A log pg (a|A)

B. Experimental Settings&RQGLWLRQV DQG SDUDPHWHUV RI WKH PHWKRGV GHVFULEHG LQ
WKH 0(7+2'6 VHFWLRQ DUH GHVFULEHG KHUH LQ GHWDLO199

FHQWURLGV 7KLV k LV VDPH QXPEHU XVHG LQ RXU SUHYLRXV ZRUN
>@ 7KH QXPEHU RI FOXVWHUV XVHG WR TXDQWL]H WKH PXVLFDOWLPEUH DQG UK\WKP IHDWXUHV ZDV VHW WR  LQ WKLV HYDOXDWLRQ
3) Chord estimation: :LWK 6RQJOH FKRUGV DUH WUDQVFULEHG
XVLQJ  FKRUG W\SHV PDMRU PDMRU WK PDMRU WK GRPLQDQW WK PLQRU PLQRU WK KDOIGLPLQLVKHG GLPLQLVKHGDXJPHQWHG DQG ¿YH YDULDQWV RI PDMRU FKRUGV ZLWK GLIIHUHQWEDVV QRWHV    E DQG   7KH UHVXOWLQJ  FKRUGV W\SHV ×  URRW QRWHV DQG RQH ³QR FKRUG´ ODEHO DUH
HVWLPDWHG VHH >@ IRU GHWDLOV 4) Training the generative models: 7UDLQLQJ VRQJ PRGHOVDQG VRQJVHW PRGHOV RI WKH  PXVLFDO HOHPHQWV E\ /'$ DQG
93</0 ZH XVHG DOO RI WKH  RULJLQDO UHFRUGLQJV RI WKH-323 0'%
7KH QXPEHU RI WRSLFV K ZDV VHW WR 100 DQG WKH PRGHOSDUDPHWHUV RI /'$ ZHUH WUDLQHG E\ XVLQJ WKH FROODSVHG*LEEV VDPSOHU >@ 7KH K\SHUSDUDPHWHUV RI WKH 'LULFKOHWGLVWULEXWLRQV IRU WRSLFV DQG ZRUGV ZHUH LQLWLDOO\ VHW WR 1 DQG
0.1 UHVSHFWLYHO\ 7KH FRQGLWLRQV ZHUH EDVHG RQ RXU SUHYLRXVZRUN >@
7KH QXPEHU RI FKRUGV XVHG WR PRGHO FKRUG SURJUHVVLRQZDV  WKH  FKRUG W\SHV PDMRU PDMRU WK PDMRU WKGRPLQDQW WK PLQRU PLQRU WK GLPLQLVKHG DXJPHQWHG IRUHDFK RI WKH  GLIIHUHQW URRW QRWHV DQG RQH ³QR FKRUG´ ODEHO
97 = 8 × 12 + 1 5) Baseline methods: $V EDVHOLQH PHWKRGV VLPSOH PHWKRGV ZHUH XVHG WR HVWLPDWH WKH VLPLODULW\ DQG FRPPRQQHVV
7KH EDVHOLQH PHWKRGV XVHG WR HVWLPDWH WKH VLPLODULW\RI YRFDO WLPEUH PXVLFDO WLPEUH DQG UK\WKP FDOFXODWHG
WKH (XFOLGHDQ GLVWDQFH EHWZHHQ PHDQ IHDWXUH YHFWRUV RIWZR VRQJV ,Q WKH EDVHOLQH PHWKRGV XVHG WR HVWLPDWH WKH
FRPPRQQHVV RI WKHVH HOHPHQWV WKH PHDQ IHDWXUH YHFWRUVZHUH FDOFXODWHG IRU D VRQJVHW ZHUH XVHG WR FDOFXODWH WKH(XFOLGHDQ GLVWDQFH IURP D WDUJHW VRQJ (DFK PHDQ YHFWRUZDV QRUPDOL]HG E\ VXEWUDFWLQJ WKH PHDQ DQG GLYLGLQJ E\
WKH VWDQGDUG GHYLDWLRQ
7R PRGHO FKRUG SURJUHVVLRQ ZH XVHG DV D VRQJ PRGHO D
XQLJUDP PRGHO WUDLQHG E\ PD[LPXP OLNHOLKRRG HVWLPDWLRQ
7KH EDVHOLQH PRGHOLQJ RI FKRUG SURJUHVVLRQ RI D VHW RI PXVLFDO SLHFHV WKH +3</0 nJUDP PRGHO >@ DV D VRQJVHWPRGHO n LV VHW WR 1  7R DYRLG WKH ]HURIUHTXHQF\ SUREOHPFKRUG VLPLODULW\ EHWZHHQ WZR VRQJV LV DOVR HVWLPDWHG E\
FDOFXODWLQJ ZHLJKWHG PHDQ SUREDELOLWLHV RI WKH VRQJ PRGHODQG WKH VRQJVHW PRGHO 7KH ZHLJKWV DUH (1 − r) DQG r
UHVSHFWLYHO\ r LV VHW WR 10−5 

7DEOH ,
6 ,1*(56 2) 7+(  621*6 86(' ,1 7+( (;3(5,0(176 ,'$UWLVW QDPH

$
%
&
'
(
)
*
+
,
.
/
0
1
2
3
4
5
6
7

$\XPL +DPDVDNL%¶]
0RUQLQJ 0XVXPH
0DL .XUDNL.XPL .RGD%R$(;,/(/¶$UFHQ&LHO
5LQD $LXFKLZLQGV
623+,$
0LND 1DNDVKLPD&+(0,675<*DFNW*$51(7 &52:
72.,2
3RUQR *UDI¿WWL.HQ +LUDL(YHU\ /LWWOH 7KLQJ
*/$<
7RWDO

*HQGHU RI YRFDOLOVW V
 PRUH WKDQ RQH VLQJHU
IHPDOHPDOH
IHPDOHIHPDOHIHPDOHIHPDOHPDOH
PDOH
IHPDOHPDOH
PDOH
IHPDOHPDOH
PDOH
IHPDOH
PDOH
PDOH
PDOH
IHPDOHPDOH
PDOH  ± IHPDOH 

1XPEHU RI VRQJV






















1) Extracting acoustic features: )RU YRFDO WLPEUH IHDWXUHV ZH WDUJHWHG PRQDXUDO N+] GLJLWDO UHFRUGLQJV DQG
H[WUDFWHG ΔF0 DQG WKRUGHU /30&&V HYHU\  PV 7KH
IHDWXUH YHFWRUV ZHUH H[WUDFWHG IURP HDFK VRQJ XVLQJ DVUHOLDEOH YRFDO IUDPHV WKH WRS  RI WKH IHDWXUH IUDPHV8VLQJ WKH  VRQJV RI WKH 5:& 0'% D YRFDO *00
DQG D QRQYRFDO *00 ZHUH WUDLQHG E\ YDULDWLRQDO %D\HVLDQLQIHUHQFH >@ :H VHW WKH QXPEHU RI *DXVVLDQV WR  DQG
VHW WKH K\SHUSDUDPHWHU RI D 'LULFKOHW GLVWULEXWLRQ RYHU WKH
PL[LQJ FRHI¿FLHQWV WR  )RU PXVLFDO WLPEUH IHDWXUHV ZH WDUJHWHG PRQDXUDO N+] GLJLWDO UHFRUGLQJV DQG H[WUDFWHG ΔSRZHU WKRUGHU0)&&V DQG WKRUGHU Δ0)&&V HYHU\  PV 7KH IHDWXUHYHFWRUV ZHUH H[WUDFWHG IURP  RI WKH IUDPHV RI HDFK VRQJ
DQG WKRVH IUDPHV ZHUH VHOHFWHG UDQGRPO\)RU UK\WKPEDVHG IHDWXUHV ZH WDUJHWHG PRQDXUDO 
N+] GLJLWDO UHFRUGLQJV DQG H[WUDFWHG )3V E\ XVLQJ WKH 0XVLF$QDO\VLV 0$ WRROER[ IRU 0DWODE >@ $ GLPHQVLRQ)3 YHFWRU ZDV HVWLPDWHG HYHU\  VHFRQGV DQG WKH DQDO\VLVIUDPH OHQJWK ZDV  VHFRQGV :H WKHQ UHGXFHG WKH QXPEHU RIYHFWRU GLPHQVLRQV E\ XVLQJ 3&$ EDVHG RQ WKH FXPXODWLYH
FRQWULEXWLRQ UDWLR ≤ 95%  $ SURMHFWLRQ PDWUL[ IRU 3&$ZDV FRPSXWHG E\ XVLQJ WKH  VRQJV RI WKH 5:& 0'%)LQDOO\ D GLPHQVLRQDO SURMHFWLRQ PDWUL[ ZDV REWDLQHG
7KH FRQGLWLRQV GHVFULEHG DERYH e.g. WKH  DQG 
N+] VDPSOLQJ IUHTXHQFLHV ZHUH EDVHG RQ SUHYLRXV ZRUNV
2) Quantization: 7R TXDQWL]H WKH YRFDO IHDWXUHV ZH VHWWKH QXPEHU RI FOXVWHUV RI WKH kPHDQV DOJRULWKP WR  DQG
DOVR WUDLQHG XVHG WKH  VRQJV RI WKH 5:& 0%' WR WUDLQ WKH

C. Experiment A: Similarity estimation
7R HYDOXDWH PXVLFDO VLPLODULW\ HVWLPDWLRQ EDVHG RQ SUREDELOLVWLF JHQHUDWLYH PRGHOV H[SHULPHQW $ XVHG DOO  VRQJVIRU PRGHOLQJ DQG WKH  VRQJV E\ WKH DUWLVWV OLVWHG LQ 7DEOH, DA  7KH  VRQJV ZHUH VXQJ E\ WKH WZHQW\ DUWLVWVZLWK WKH JUHDWHVW QXPEHU RI VRQJV LQ WKH PRGHOLQJ VHW 7KH
HYDOXDWLRQ VHW LV YHU\ GLYHUVH DUWLVWV LQFOXGH VROR VLQJHUV DQG

 7KH WUDLQHG *00V ZHUH PRGHOV LQ ZKLFK WKH QXPEHU RI *DXVVLDQV ZDVUHGXFHG WR  IRU WKH YRFDO *00 DQG WR  IRU WKH QRQYRFDO *00200

songs (target of computing similarity)
songs (target of computing similarity)(a) vocal timbre

7DEOH ,,
7 +( 7:(17< 621* 3$,56 %(/21* 72 7:2 *52836 (d) musical timbre

450
400

9RFDO
0XVLFDO&KRUG
5K\WKPWLPEUHWLPEUHSURJUHVVLRQ
/±2
%±$
.±*
'±,
)±6
+±7
,±6
2±%
-±,
4±.
'±&
.±1
$±'
0±5
(±4
$±%±4
'±/
$±)
+±7
WRS
0±5
6±,
2±+
3±&
+±3
3±1
5±/
6±/
(±&
2±*
1±7
)±(
*±1
(±)
3±%
4±*
.±7
-±&
-±0
0±5
)±(
*±3±2
2±3
7±2±(
+±&
7±5
+±'
&±%
*±6
%±0
3±$
7±5
1±(
4±1
4
±
/
4
±
$
0
±
4
-±$
ERWWRP
2±%
3±)
%±5
'±.
*±6
,±0
.±)
6±*
&±1
6±1
/±'
+±)
0±.
+±/
7±,±&
5±,
.±'
$±,
/±(
/ ± 2 IRU H[DPSOH PHDQV D VRQJ RI VLQJHU / DQG D VRQJ RI VLQJHU 2

350

*URXS

300
250
200
150
100
50

(c) rhythm(d) chord progression

450
400
350
300
250
200
150
100
50

100
200
300
400
100
200
300
400
artist ID
artist ID
A B C D E F G H I J K L MN O P QR S T
A B C D E F G H I J K L MN O P QR S T

songs (a song-specific model as query) songs (a song-specific model as query)songs (target of computing similarity)

songs (target of computing similarity))LJXUH (a) vocal timbre

7DEOH ,,,
7 +( 7:(17< 621* 3$,56 %(/21* 72 7:2 *52836 %$6(/,1( 

6LPLODULWLHV DPRQJ DOO  VRQJV(d) musical timbre

*URXS

450
400
350
300
250

WRS

200
150
100
50

(c) rhythm(d) chord progression

450
400

ERWWRP

350
300
250
200
150

9RFDO
WLPEUH
)±(
$±&
/±,
1±.
5±*
7±4
%±+
'±2
0±3±6
&±7
+±6
.±%
3±)
(±0
1±'
$±/±4
*±,
2±5

0XVLFDOWLPEUH
)±(
0±,
3±%±6
.±'
$±2
+±1
7±&
*±5
/±4
3±2
6±7
4±%
+±'
,±*
0±.
$±/
1±&
5±(
)±-

5K\WKP
1±/
2±(
%±,
5±*
&±$
7±.
6±3
)±'
0±+
4±1±7
/±+
2±%
(±$
6±3
&±)
-±.
5±0
*±,
'±4

&KRUG
SURJUHVVLRQ
)±(
0±,
3±%±6
.±'
$±2
+±1
7±&
*±5
/±4
3±2
6±7
4±%
+±'
,±*
0±.
$±/
1±&
5±(
)±-

100
50

PDWUL[ IRU HDFK RI WKHVH HOHPHQWV DQG )LJXUH  VKRZV WKHEDVHOLQH UHVXOWV ,Q HDFK ¿JXUH WKH KRUL]RQWDO D[LV LQGLFDWHV
TXHU\ VRQJ PRGHOV DQG WKH YHUWLFDO D[LV LQGLFDWHV WDUJHWVRQJV RI FRPSXWLQJ VLPLODULW\
7KH VLPLODULW\ PDWUL[ UHSUHVHQWV 214, 369 = 463 × 463
SDLUV ,Q WKH PDWULFHV RQO\ WKH  WDUJHW VRQJV  RI
DA KDYLQJ WKH KLJKHVW VLPLODULWLHV IRU HDFK RI WKH TXHULHVDUH FRORUHG EODFN2) Comparing estimated similarities with human ratings::H QH[W HYDOXDWHG WKH VRQJ PRGHOV E\ XVLQJ KXPDQ UDWLQJV
7ZHQW\ VRQJ SDLUV EHORQJHG WR WZR JURXSV UHIHUUHG WR DV

100
200
300
400
100
200
300
400
artist ID
artist ID
A B C D E F G H I J K L MN O P QR S T
A B C D E F G H I J K L MN O P QR S T

songs (a song-specific model as query) songs (a song-specific model as query))LJXUH %DVHOLQH VLPLODULWLHV DPRQJ DOO  VRQJVEDQGV DQG D EDODQFH RI PDOH DQG IHPDOH YRFDOLVWV  DQG
 UHVSHFWLYHO\ 1) Similarity matrix: :H ¿UVW HVWLPDWHG WKH VLPLODULWLHV
EHWZHHQ WKH  VRQJV ZLWK UHVSHFW WR WKH IRXU PXVLFDO
HOHPHQWV )LJXUHV  D WKURXJK G VKRZ WKH VLPLODULW\

201

(a) vocal timbre

topic distribution (rhythm)(b) musical timbre
0.2

mixing weightsimilarity rating7

p < 0.1%

p < 0.1%

4
1
7

0
10

4

20

30

40

50

60

70

90 topic80)LJXUH  7RS WRSLF GLVWULEXWLRQ RI D VRQJ WKDW JHWV KLJK VLPLODULW\ ZLWKPRVW VRQJV %RWWRP WRSLF GLVWULEXWLRQ RI D VRQJ WKDW JHWV ORZ VLPLODULW\ZLWK PRVW VRQJV

1bottom10

top10

bottom10
similarity(chord progression)

top10)LJXUH  %R[ SORWV VKRZLQJ WKH VWDWLVWLFV IRU WKH VRQJSDLU VLPLODULW\
UDWLQJV E\ D PXVLFLDQ(a) vocal timbre
7(b) musical timbre

p < 1%

query (singer A)1(d) chord progression
p < 10%p < 0.1%

4

bottom10

100

150

200

250

300

350

400

song ID0query (singer A)

top10

bottom10

query (singer F)

-5
-10
-15
0

1
top10

50

query (singer F))LJXUH  %OXH VLPLODULW\ EHWZHHQ D TXHU\ VRQJ DQG RWKHUV ZLWK KLJKVLPLODULW\ LQ PRVW FDVHV 5HG VLPLODULW\ EHWZHHQ D TXHU\ VRQJ DQG RWKHUVZLWK ORZ VLPLODULW\ LQ PRVW FDVHV

4(c) rhythm

0
-2
-4
-6
-8
-10
-12
0

baseline similarity(chord progression)

similarity rating

a song of artist D

0.1

p < 5%

p < 1%

700.2(d) chord progression(c) rhythma song of artist T

0.1

50

100

150

200

250

300

350

400

song ID)LJXUH  %OXH EDVHOLQH VLPLODULW\ EHWZHHQ D TXHU\ VRQJ DQG RWKHUV 5HGEDVHOLQH VLPLODULW\ EHWZHHQ D TXHU\ VRQJ DQG RWKHUV ZLWK ORZ VLPLODULW\ LQ
PRVW FDVHV)LJXUH %R[ SORWV VKRZLQJ WKH VWDWLVWLFV IRU WKH EDVHOLQH VRQJSDLU
VLPLODULW\ UDWLQJVWLPEUH WKH  OHYHO IRU UK\WKP DQG WKH  OHYHO IRU FKRUG
SURJUHVVLRQ )LJXUH  3) Discussion: )URP WKH VLPLODULW\ PDWUL[ RQH VHHV WKDWVRQJV E\ WKH VDPH DUWLVW KDYH KLJK VLPLODULW\ IRU YRFDO WLPEUH
DQG PXVLFDO WLPEUH )RU UK\WKP DQG FKRUG SURJUHVVLRQ RQ
WKH RWKHU KDQG VRPH VRQJV E\ WKH VDPH DUWLVW KDYH KLJKVLPLODULW\ LQGLFDWHG E\ DUURZV LQ )LJXUHV  F DQG G
EXW PRVW GR QRW 7KHVH UHVXOWV UHÀHFW PXVLFDO FKDUDFWHULVWLFVTXDOLWDWLYHO\ DQG FDQ EH XQGHUVWRRG LQWXLWLYHO\2Q WKH VLPLODULW\ PDWUL[ IRU UK\WKP KRUL]RQWDO OLQHV FDQ
EH VHHQ 7KLV PHDQV WKDW WKHUH DUH VRQJV WKDW LQ PRVW FDVHV
JHW KLJK VLPLODULW\ UHJDUGOHVV RI ZKLFK VRQJ LV WKH TXHU\
VRQJ 2Q WKH RWKHU KDQG WKHUH DUH DOVR VRQJV WKDW JHW ORZ
VLPLODULW\ ZLWK PRVW TXHU\ VRQJV /'$ WRSLF GLVWULEXWLRQVIRU ERWK NLQGV DUH VKRZQ LQ )LJXUH  7KH IRUPHU NLQG¶V LV
ÀDW DQG KDV VRPH WRSLFV KDYLQJ YDOXH DQG WKH ODWWHU NLQG¶V
KDV D IHZ WRSLFV KDYLQJ YDOXH 2Q WKH VLPLODULW\ PDWUL[ IRUFKRUG SURJUHVVLRQ E\ XVLQJ WKH WULJUDP VRQJ PRGHOV WKHUH
DUH TXHU\ VRQJV WKDW JHW KLJK VLPLODULW\ ZLWK DOO RWKHU VRQJV
e.g. D VRQJ RI VLQJHU $ DQG WKHUH DUH TXHU\ VRQJV WKDWJHW ORZ VLPLODULW\ ZLWK DOO RWKHU VHH IRU H[DPSOH )LJXUH  ,Q WKH EDVHOLQH XQLJUDP VHWWLQJ RQ WKH RWKHU KDQG WKH
TXHU\ VRQJ RI VLQJHU $ KDV GLIIHUHQW VLPLODULWLHV ZLWK DOO
RWKHU VRQJV )LJXUH  

WKH top10 DQG bottom10 7KH WRS JURXS LQFOXGHG WKH
WHQ VRQJ SDLUV KDYLQJ WKH KLJKHVW VLPLODULWLHV IRU HDFK RIWKH PXVLFDO HOHPHQWV XQGHU WKH VHOHFWLRQ UHVWULFWLRQ WKDWWKHUH LV QR RYHUODSSLQJ RI VLQJHU QDPHV LQ WKH JURXS 7KLVPHDQV WKDW WKLV JURXS FRPSULVHV RQO\ SDLUV RI VRQJV VXQJ
E\ GLIIHUHQW VLQJHUV 7KH ERWWRP JURXS LQFOXGHV WKH WHQ
VRQJ SDLUV DOVR VHOHFWHG XQGHU WKH QRRYHUODSSLQJQDPH
FRQGLWLRQ KDYLQJ WKH ORZHVW VLPLODULWLHV IRU HDFK RI WKH
PXVLFDO HOHPHQWV 7DEOH ,,, VKRZV WKH WRS DQG ERWWRPJURXSV$ PDOH PXVLFLDQ ZKR KDG H[SHULHQFH ZLWK DXGLR PL[LQJPDVWHULQJ DQG DUUDQJHPHQWFRPSRVLWLRQ RI -DSDQHVH
SRSXODU VRQJV ZDV DVNHG WR UDWH VRQJSDLU VLPLODULW\ RQ D 
SRLQW VFDOH UDQJLQJ IURP  QRW VLPLODU WR  YHU\ VLPLODU 
5DWLQJ WR D SUHFLVLRQ RI RQH GHFLPDO SODFH e.g.  ZDVDOORZHG)LJXUH  VKRZV WKH UHVXOWV RI WKH UDWLQJ DQG )LJXUH  VKRZVWKH UHVXOWV RI WKH UDWLQJ EDVHG RQ WKH EDVHOLQH UHVXOWV 7KH
VWDWLVWLFV RI WKH UDWLQJV DUH VKRZQ E\ ER[ SORWV LQGLFDWLQJPHGLDQ YDOXHV  TXDQWLOHV  TXDQWLOHV PLQLPXP YDOXHVDQG PD[LPXP YDOXHV 7HVWLQJ WKH UHVXOWV E\ XVLQJ :HOFK¶V t
WHVW >@ UHYHDOHG WKDW WKH GLIIHUHQFHV EHWZHHQ WKH WZR JURXSVZHUH VLJQL¿FDQW DW WKH  OHYHO IRU YRFDO DQG PXVLFDO

202

7DEOH ,9
3 ($5621 352'8&7 020(17 &255(/$7,21 &2()),&,(176 %(7:((1The number of songs

similarity

110(67,0$7(' &200211(66 2) 7+( )285 (/(0(176 ($&+ 621* $1'
7+( 180%(5 2) 621*6 +$9,1* +,*+ 6,0,/$5,7< :,7+ 7+( 621* 

-4.23
-7
1000

the number of songs having high similarity

1861

-3

2000

(* 0.766)

110
0
-5

(c) rhythm(* 0.735)

0

1000

-7

2000

song ID(b) musical timbre(a) vocal timbre

2000

4000

3000

3/4
quantile
value

3000

1861

(* 0.834)

0
-3.4

-4.5
4000

-3.7

(d) chord progression(* 0.677)

FRUUHODWLRQ FRHI¿FLHQWV

HOHPHQWFRQGLWLRQ

&

&%

YRFDO
PXVLFDO WLPEUH
UK\WKPFKRUG SURJUHVVLRQYRFDOPXVLFDO WLPEUH
UK\WKPFKRUG SURJUHVVLRQ

6
6
6
6
6%
6%
6%
6%










−








&RQGLWLRQV
6 7KH QXPEHU RI VRQJV KDYLQJ KLJK VLPLODULW\
6% 7KH QXPEHU RI VRQJV KDYLQJ KLJK VLPLODULW\ EDVHOLQH& FRPPRQQHVV&% FRPPRQQHVV EDVHOLQH

0
-4
0
-2
commonness
commonness(*x): Pearson product-moment correlation coefficient(VWLPDWHG VLPLODULW\ LV FRPSDUDEOH WR UDWLQJV E\ D PXVLFLDQ DW WKH  VLJQL¿FDQFH OHYHO )LJXUH  DQG  DW WKH  VLJQL¿FDQFH OHYHO )LJXUH  DQG )LJXUH 
5HODWLRQVKLSV EHWZHHQ HVWLPDWHG FRPPRQQHVV RI WKH IRXU
HOHPHQWV HDFK VRQJ DQG WKH QXPEHU RI VRQJV KDYLQJ KLJK VLPLODULW\ ZLWKWKH VRQJ

8QGHU FRQGLWLRQV RI WKH UHODWLYHO\ UHOLDEOH VLPLODULWLHV³YRFDO 6 ´ ³PXVLFDO WLPEUH 6 ´ DQG ³UK\WKP 6 ´
WKH YDOXHV RI WKH FRUUHODWLRQ FRHI¿FLHQW RI WKH SURSRVHG
PHWKRG ³&´   DQG  LV ELJJHU WKDQ WKHEDVHOLQH PHWKRG ³&%´ −  DQG   7KH
UHVXOWV VXJJHVW WKDW WKH PRUH VLPLODU D VRQJ LV WR VRQJV RI WKH
VRQJ VHW WKH KLJKHU LWV PXVLFDO FRPPRQQHVV LQ WKH SURSRVHG
PHWKRG $OWKRXJK WZR FRHI¿FLHQWV RI WKH FRQGLWLRQ ³YRFDO
6% ´ DQG ³FKRUG SURJUHVVLRQ 6% ´ DUH SRVLWLYH YDOXH ³&´ DQG   WKH YDOXH RI WKH EDVHOLQH PHWKRG ³&%´ DQG  LV ELJJHU WKDQ WKRVH FRHI¿FLHQWV 7KH
LPSURYHPHQW RI WKH FRUUHODWLRQ FRHI¿FLHQWV LV D VXEMHFW IRUIXWXUH GLUHFWLRQ
2) Application of commonness in terms of vocal timbre:
2QO\ WKH VRQJVHW PRGHOV RI YRFDO WLPEUH FDQ EH HYDOXDWHG
TXDQWLWDWLYHO\ E\ XVLQJ WKH VLQJHU¶V JHQGHU 7KHVH PRGHOV DUH
LQWHJUDWHG VRQJ PRGHOV ZLWK GLIIHUHQW UDWLRV RI WKH QXPEHURI PDOH VLQJHUV WR IHPDOH VLQJHUV
7R WUDLQ VRQJVHW PRGHOV ZH XVHG  VRQJV E\ GLIIHUHQWVROR VLQJHUV  PDOH DQG  IHPDOH IURP WKH -323 0'%:H WUDLQHG WKUHH W\SHV RI VRQJVHW PRGHOV RQH WUDLQHG E\
XVLQJ DOO  VRQJV RQH WUDLQHG E\ XVLQJ RQH IHPDOH VRQJ
DQG DOO  PDOH VRQJV DQG RQH LV WUDLQHG E\ XVLQJ RQH PDOH
VRQJ DQG DOO  PDOH VRQJV)LJXUH  VKRZV WKH YRFDO WLPEUH FRPPRQQHVVHV EDVHG
RQ WKH  GLIIHUHQW VRQJVHW PRGHOV :KHQ D PRGHO ZLWK DKLJK SURSRUWLRQ RI IHPDOH VRQJV LV XVHG WKH FRPPRQQHVVRI VRQJV VXQJ E\ IHPDOHV LV KLJKHU WKDQ WKH FRPPRQQHVVRI VRQJV VXQJ E\ PDOHV DQG YLFH YHUVD  )LJXUH  VKRZVWKH VWDWLVWLFV RI WKH FRPPRQQHVVHV DUH VKRZQ E\ ER[ SORWV
7KH UHVXOWV VXJJHVW WKH FRPPRQQHVVHV FDQ EH UHÀHFWHG YRFDOWUDFW IHDWXUHV

7KH FRPSDULVRQ ZLWK WKH UHVXOWV RI WKH H[SHUW UDWLQJV
VXJJHVWV WKDW WKH SURSRVHG PHWKRGV FDQ HVWLPDWH PXVLFDO
VLPLODULW\ DSSURSULDWHO\ 7R LPSURYH WKH SHUIRUPDQFH ZLWKUHJDUG WR DOO HOHPHQWV FRQGLWLRQV VXFK DV WKRVH IRU H[WUDFWLQJDFRXVWLF IHDWXUHV IRU TXDQWL]DWLRQ IRU FKRUG HVWLPDWLRQ DQG
IRU PRGHO WUDLQLQJ FDQ EH FRQVLGHUHG LQ IXWXUH ZRUN
D. Experiment B: Commonness estimation
7R HYDOXDWH PXVLFDO FRPPRQQHVV HVWLPDWLRQ EDVHG RQ
SUREDELOLVWLF JHQHUDWLYH PRGHOV H[SHULPHQW % DOVR XVHG WKH VRQJV RI WKH -323 0'% WR WUDLQ WKH VRQJVHW PRGHOVDQG IRU HYDOXDWLQJ HDFK PXVLFDO HOHPHQW
:KHQ HYDOXDWLQJ WKH FRPPRQQHVV HVWLPDWLRQ PHWKRG ZH¿UVW HYDOXDWHG WKH QXPEHU RI VRQJV KDYLQJ KLJK VLPLODULW\)RU H[DPSOH LQ )LJXUH  WKH VRQJ a KDV PDQ\ VLPLODU
VRQJV LQ WKH VRQJ VHW A ,I D VRQJ KDYLQJ KLJKHU PXVLFDOFRPPRQQHVV DQG LW LV YHU\ VLPLODU WR VRQJV RI D VRQJ VHW1) Relation between commonness and the number ofsongs having high similarity: )LJXUH  VKRZV WKH UHODWLRQVKLSV FRPSDULQJ WKH HVWLPDWHG FRPPRQQHVV RI VRQJVFRQWDLQHG LQ WKH -323 0'% WR WKH QXPEHU RI VRQJV KDYLQJKLJK VLPLODULW\ :H XVHG DV WKH WKUHVKROG IRU GHFLGLQJ WKH
VLPLODULW\ RI DQ HOHPHQW WR EH KLJK WKH  TXDQWLOH YDOXHRI DOO VLPLODULWLHV DPRQJ DOO SRVVLEOH VRQJSDLUV LQ WKH -323
0'%  = 3278 × 3278 VRQJSDLUV 
7KH 3HDUVRQ SURGXFWPRPHQW FRUUHODWLRQ FRHI¿FLHQWV DUH
DOVR VKRZQ LQ HDFK SDUW RI WKH ¿JXUH DQG 7DEOH ,9 7KH
UHOLDELOLW\ RI WKH HVWLPDWHG VLPLODULW\ FDQ EH HYDOXDWHG E\
XVLQJ WKH UHVXOWV RI )LJXUH  DQG  7KH DVWHULVN PDUN
DQG WKH GRXEOHDVWHULVN PDUNLQGLFDWH WKH GLIIHUHQFHV
EHWZHHQ WKH WZR JURXSV WKH WRS DQG ERWWRP JURXSVVLJQL¿FDQW DW WKH  DQG  OHYHO UHVSHFWLYHO\203

commonnessbaseline1

3 song-set models:

2

3

4

5

6

5 ()(5(1&(6female vocalists

male vocalists

7 songs

0.03

7B, H, N, Q, R, T, A

14 songs
9 songs 1 7 8 9 10 11 12 13 14
ALL
B, A, D, E, F, I, L, O, S

>@ 0 *RWR DQG . +LUDWD ³5HFHQW VWXGLHV RQ PXVLF LQIRUPDWLRQ SURFHVVLQJ´ Acoustical Science and Technology (edited by the AcousticalSociety of Japan) YRO  QR  SS ± 
>@ 3 .QHHV DQG 0 6FKHGO ³$ VXUYH\ RI PXVLF VLPLODULW\ DQG UHFRPPHQGDWLRQ IURP PXVLF FRQWH[W GDWD´ ACM Trans. on Multimedia
Computing, Communications and Applications YRO  QR  SS ± 
>@ % 3DUGR (G Special issue: Music information retrieval VHU &RPPXQLFDWLRQV RI WKH $&0  YRO  QR  SS ±>@ 0 &DVH\ 5 9HOWNDPS 0 *RWR 0 /HPDQ & 5KRGHV DQG
0 6ODQH\ ³&RQWHQWEDVHG PXVLF LQIRUPDWLRQ UHWULHYDO &XUUHQW GLUHF
WLRQV DQG IXWXUH FKDOOHQJHV´ Proc. IEEE YRO  QR  SS ±>@ - 6 'RZQLH ³7KH PXVLF LQIRUPDWLRQ UHWULHYDO HYDOXDWLRQ H[FKDQJH
±  $ ZLQGRZ LQWR PXVLF LQIRUPDWLRQ UHWULHYDO UHVHDUFK´
Acoust. Sci. & Tech. YRO  SS ± 
>@ - 6 'RZQLH ' %\UG DQG 7 &UDZIRUG ³7HQ \HDUV RI ,60,5
5HÀHFWLRQV RQ FKDOOHQJHV DQG RSSRUWXQLWLHV´ LQ Proc. ISMIR 2009 SS ±>@ < 6RQJ 6 'L[RQ DQG 0 3HDUFH ³$ VXUYH\ RI PXVLF UHFRPPHQGDWLRQ V\VWHPV DQG IXWXUH SHUVSHFWLYHV´ LQ Proc. CMMR 2012 
SS ±>@ / : %DUVDORX ³,GHDOV FHQWUDO WHQGHQF\ DQG IUHTXHQF\ RI LQVWDQWLDWLRQ DV GHWHUPLQDQWV RI JUDGHG VWUXFWXUH LQ FDWHJRULHV´ journal of
Experimental Psychology: Learning, Memory, and Cognition YRO 
QR  SS ± 
>@ -- $XFRXWXULHU DQG ) 3DFKHW ³0XVLF VLPLODULW\ PHDVXUHV :KDW¶V
WKH XVH"´ LQ Proc. ISMIR 2002  SS ±>@ + )XMLKDUD 0 *RWR 7 .LWDKDUD DQG + * 2NXQR ³$ PRGHOLQJRI VLQJLQJ YRLFH UREXVW WR DFFRPSDQLPHQW VRXQGV DQG LWV DSSOLFDWLRQWR VLQJHU LGHQWL¿FDWLRQ DQG YRFDOWLPEUHVLPLODULW\ EDVHG PXVLF LQIRUPDWLRQ UHWULHYDO´ IEEE Trans. on ASLP YRO  QR  SS ±>@ ' 0 %OHL $ < 1J DQG 0 , -RUGDQ ³/DWHQW 'LULFKOHW DOORFDWLRQ´
Journal of Machine Learning Research YRO  SS ± 
>@ ' 0RFKLKDVKL DQG ( 6XPLWD ³,Q¿QLWH 0DUNRY PRGHO´ LQ Proc.
NIPS2007 
>@ . <RVKLL DQG 0 *RWR ³$ YRFDEXODU\IUHH LQ¿QLW\JUDP PRGHO IRU
QRQSDUDPHWULF ED\HVLDQ FKRUG SURJUHVVLRQ DQDO\VLV´ LQ Proc. ISMIR
2011  SS ±>@ 7 1DNDQR . <RVKLL DQG 0 *RWR ³9RFDO WLPEUH DQDO\VLV XVLQJODWHQW 'LULFKOHW DOORFDWLRQ DQG FURVVJHQGHU YRFDO WLPEUH VLPLODULW\´
LQ Proc. ICASSP 2014  SS ±
>@ 0 *RWR ³$ UHDOWLPH PXVLF VFHQH GHVFULSWLRQ V\VWHP 3UHGRPLQDQW) HVWLPDWLRQ IRU GHWHFWLQJ PHORG\ DQG EDVV OLQHV LQ UHDOZRUOG DXGLR
VLJQDOV´ Speech Communication YRO  QR  SS ± 
>@ ( 3DPSDON Computational models of music similarity and theirapplication to music information retrieval
3K' 'LVVHUWDWLRQ
9LHQQD ,QVW RI 7HFK 
>@ ( 3DPSDON $ 5DXEHU DQG ' 0HUNO ³&RQWHQWEDVHG RUJDQL]DWLRQ
DQG YLVXDOL]DWLRQ RI PXVLF DUFKLYHV´ LQ Proc. ACMMM ’02 
SS ±>@ 0 0DXFK DQG 0 /HY\ ³6WUXFWXUDO FKDQJH RQ PXOWLSOH WLPH VFDOHVDV D FRUUHODWH RI PXVLFDO FRPSOH[LW\´ LQ Proc. ISMIR 2011 
SS ±>@ 0 *RWR . <RVKLL + )XMLKDUD 0 0DXFK DQG 7 1DNDQR ³6RQJOH
$ ZHE VHUYLFH IRU DFWLYH PXVLF OLVWHQLQJ LPSURYHG E\ XVHU FRQWULEXWLRQV´ LQ Proc. ISMIR 2011  SS ±>@ 0 *RWR + +DVKLJXFKL 7 1LVKLPXUD DQG 5 2ND ³5:& PXVLF
GDWDEDVH 3RSXODU FODVVLFDO DQG MD]] PXVLF GDWDEDVHV´ LQ Proc. ISMIR
2002  SS ±>@ & 0 %LVKRS Pattern recognition and machine learning 6SULQJHU
9HUODJ 1HZ <RUN ,QF 
>@ 7 / *ULI¿WKV DQG 0 6WH\YHUV ³)LQGLQJ VFLHQWL¿F WRSLFV´ LQ Proc.
Natl. Acad. Sci. USA (PNAS) YRO   SS ±
>@ < : 7HK ³$ KLHUDUFKLFDO ED\HVLDQ ODQJXDJH PRGHO EDVHG RQ 3LWPDQ<RU SURFHVVHV´ LQ Proc. COLING/ACL 2006  SS ±>@ % / :HOFK ³7KH VLJQL¿FDQFH RI WKH GLIIHUHQFH EHWZHHQ WZR PHDQVZKHQ WKH SRSXODWLRQ YDULDQFHV DUH XQHTXDO´ Biometrika YRO  SS
± 

0.02
0.01
0
0.4

0

1

2

3

4

5

6

7

8

9

10

11

12

13

B

H

N

Q

R

T

A

D

E

F

I

L

O

male vocalists

14 songs
S artist ID

female vocalists)LJXUH  9RFDO WLPEUH FRPPRQQHVV EDVHG RQ  GLIIHUHQW VRQJVHW PRGHOVIRU  VRQJV  PDOH DQG  IHPDOH 
male vocalists

0.03

7 songs

Model 1

1

2

3

4

5

6

Model 2Model 2

9 songs 1 7 8 9 10 11 12 13 14
B, A, D, E, F, I, L, O, S

Model 1Model 2male female(1-6) (7-14)

male female(1-6) (7-14)

0.4

0.02
0.01
0

female vocalists

7

B, H, N, Q, R, T, A

baselinecommonness

Model 1

p < 1%

p < 1%

male female(1-6) (7-14)

male female(1-6) (7-14)

p < 1%0)LJXUH  %R[ SORWV VKRZLQJ WKH VWDWLVWLFV IRU WKH YRFDO WLPEUH FRPPRQQHVV )LJXUH  

,9 & 21&/86,216 $1' )8785( :25.
7KLV SDSHU GHVFULEHV D PXVLFDO VLPLODULW\ DQG FRPPRQQHVV
HVWLPDWLRQ PHWKRG EDVHG RQ SUREDELOLVWLF JHQHUDWLYH PRGHOV
/'$ DQG WKH 93</0 )RXU PXVLFDO HOHPHQWV DUH PRGHOHGYRFDO WLPEUH PXVLFDO WLPEUH UK\WKP DQG FKRUG SURJUHVVLRQ
7KH FRPPRQQHVV FDQ EH HVWLPDWHG E\ XVLQJ VRQJVHW PRGHOV ZKLFK LV HDVLHU WKDQ HVWLPDWLQJ WKH PXVLFDO VLPLODULWLHVRI DOO SRVVLEOH SDLUV RI VRQJV
7KH H[SHULPHQWDO UHVXOWV VKRZHG WKDW RXU PHWKRGV DUHDSSURSULDWH IRU HVWLPDWH PXVLFDO VLPLODULW\ DQG FRPPRQQHVV
7KH SUREDELOLW\ FDOFXODWLRQ FDQ EH DSSOLHG QRW RQO\ WR D
PXVLFDO SLHFH EXW DOVR WR D SDUW RI D PXVLFDO SLHFH 7KLVPHDQV WKDW PXVLFDO FRPPRQQHVV LV DOVR XVHIXO WR FUHDWRUVEHFDXVH D PXVLFDO HOHPHQW WKDW KDV KLJK FRPPRQQHVV LV DQ
HVWDEOLVKHG H[SUHVVLRQ DQG FDQ EH XVHG E\ DQ\RQH FUHDWLQJ
DQG SXEOLVKLQJ PXVLFDO FRQWHQW e.g. D FKRUG SURJUHVVLRQ 
6LQFH WKLV SDSHU IRFXVHG RQ WKH DERYH IRXU HOHPHQWV ZH
SODQ WR XVH PHORG\ e.g. F0 DV WKH QH[W VWHS )XWXUH ZRUNZLOO DOVR LQFOXGH WKH LQWHJUDWLRQ RI JHQHUDWLYH SUREDELOLWLHV
EDVHG RQ GLIIHUHQW PRGHOV FDOFXODWLQJ SUREDELOLWLHV RI SDUWVRI RQH VRQJ LQYHVWLJDWLQJ HIIHFWLYH IHDWXUHV DQG GHYHORSLQJ
DQ LQWHUIDFH IRU PXVLF OLVWHQLQJ RU FUHDWLRQ E\ OHYHUDJLQJPXVLFDO VLPLODULW\ DQG FRPPRQQHVV$&.12:/('*0(177KLV SDSHU XWLOL]HG WKH 5:& 0XVLF 'DWDEDVH 3RSXODU
0XVLF  7KLV ZRUN ZDV VXSSRUWHG LQ SDUW E\ &5(67 -67

204