3URFHHGLQJVRIWKH60&WK6RXQGDQG0XVLF&RPSXWLQJ&RQIHUHQFH-XO\3DGRYD,WDO\DANCEREPRODUCER: AN AUTOMATIC MASHUP MUSIC VIDEO
GENERATION SYSTEM BY REUSING DANCE VIDEO CLIPS
ON THE WEB
Tomoyasu Nakano†1
Sora Murofushi‡3
Masataka Goto†2
Shigeo Morishima ‡3†
1DWLRQDO ,QVWLWXWH RI $GYDQFHG ,QGXVWULDO 6FLHQFH DQG 7HFKQRORJ\ $,67  -DSDQ‡:DVHGD 8QLYHUVLW\ -DSDQ
1
2
3
t.nakano[at]aist.go.jp
m.goto[at]aist.go.jp
shigeo[at]waseda.jp

ABSTRACT

Original content (1st generation)
Music video:H SURSRVH D GDQFH YLGHR DXWKRULQJ V\VWHP DanceReProducer WKDW FDQ DXWRPDWLFDOO\ JHQHUDWH D GDQFH YLGHR FOLSDSSURSULDWH WR D JLYHQ SLHFH RI PXVLF E\ VHJPHQWLQJ DQG
FRQFDWHQDWLQJ H[LVWLQJ GDQFH YLGHR FOLSV ,Q WKLV SDSHUZH IRFXV RQ WKH reuse RI HYHULQFUHDVLQJ XVHUJHQHUDWHG
GDQFH YLGHR FOLSV RQ D YLGHR VKDULQJ ZHE VHUYLFH ,Q D
YLGHR FOLS FRQVLVWLQJ RI PXVLF DXGLR VLJQDOV DQG LPDJH
VHTXHQFHV YLGHR IUDPHV  WKH LPDJH VHTXHQFHV DUH RIWHQV\QFKURQL]HG ZLWK RU UHODWHG WR WKH PXVLF 6XFK UHODWLRQVKLSV DUH GLYHUVH LQ GLIIHUHQW YLGHR FOLSV EXW ZHUH QRW GHDOWZLWK E\ SUHYLRXV PHWKRGV IRU DXWRPDWLF PXVLF YLGHR JHQHUDWLRQ 2XU V\VWHP HPSOR\V PDFKLQH OHDUQLQJ DQG EHDWWUDFNLQJ WHFKQLTXHV WR PRGHO WKHVH UHODWLRQVKLSV 7R JHQHUDWH QHZ PXVLF YLGHR FOLSV VKRUW LPDJH VHTXHQFHV WKDWKDYH EHHQ SUHYLRXVO\ H[WUDFWHG IURP RWKHU PXVLF FOLSV DUH
VWUHWFKHG DQG FRQFDWHQDWHG VR WKDW WKH HPHUJLQJ LPDJH VH
TXHQFH PDWFKHV WKH UK\WKPLF VWUXFWXUH RI WKH WDUJHW VRQJ%HVLGHV DXWRPDWLFDOO\ JHQHUDWLQJ PXVLF YLGHRV 'DQFH5H
3URGXFHU RIIHUV D XVHU LQWHUIDFH LQ ZKLFK D XVHU FDQ LQWHUDFWLYHO\ FKDQJH LPDJH VHTXHQFHV MXVW E\ FKRRVLQJ GLIIHUHQW
FDQGLGDWHV 7KLV ZD\ SHRSOH ZLWK OLWWOH NQRZOHGJH RU H[SHULHQFH LQ 0$' PRYLH JHQHUDWLRQ FDQ LQWHUDFWLYHO\ FUHDWHSHUVRQDOL]HG YLGHR FOLSVMashup music videos (User-generated video clips)

Creators+

Tomoyasu

articleNakanodistributed

under

Creative Commons Attribution 3.0 Unported License,

et

al.
the
which

This

is

of

the

permits

unre-

terms

reuse

new

2nd generation

reuse

+

+

reuse

+

new

reuse

+

reuse

+

3rd generation

...

F
2011

open-access

reuse

new

+

reuse

reuse

+
Nth generation

Figure 1 *HQHUDWLRQ RI PDVKXS PXVLF YLGHRV XVHU
JHQHUDWHG PXVLF YLGHR FOLSV E\ UHXVLQJ H[LVWLQJ RULJLQDO
FRQWHQWWKH 0$' YLGHR FOLSV JHQHUDWHG E\ XVHUV FDQ EH FRQVLGHUHG
2nd generation (secondary or derivative) content )LJXUH  ,Q D 0$' YLGHR FOLS JRRG PXVLFWRLPDJH V\QFKURQL]DWLRQ ZLWK UHVSHFW WR UK\WKP LPSUHVVLRQ DQG FRQWH[W LVLPSRUWDQW$OWKRXJK LW LV HDV\ WR HQMR\ ZDWFKLQJ 0$' PRYLHV LW
LV QRW HDV\ WR JHQHUDWH WKHP EHFDXVH D FUHDWRU QHHGV WR
 VHDUFK LQ H[LVWLQJ YLGHR FOLSV IRU LPDJH VHTXHQFHVWKDW JLYH LPSUHVVLRQV DSSURSULDWH WR D JLYHQ WDUJHW PXVLFDO SLHFH  VHJPHQW DQG FRQFDWHQDWH LPDJH VHTXHQFHV WR¿W WKH WDUJHW SLHFH DQG  WLPHVWUHWFK WKH VHTXHQFHV WRPDWFK WKH WHPSR RI WKH WDUJHW SLHFH EHFDXVH H[LVWLQJ YLGHRFOLSV XVXDOO\ KDYH WHPSL GLIIHUHQW IURP WKH WHPSR RI WKH
WDUJHW SLHFH 0RUHRYHU IRU EHWWHU PXVLFWRLPDJH V\QFKURQL]DWLRQ WKH PXVLF VWUXFWXUH DQG FRQWH[W RI D PXVLFDO SLHFH
DQG LPDJH VHTXHQFHV VKRXOG EH WDNHQ LQWR DFFRXQW EXW LW
UHTXLUHV FRQVLGHUDEOH WLPH DQG HIIRUW
7R JLYH D FKDQFH RI HQMR\LQJ VXFK GLI¿FXOW 0$' PRYLH
JHQHUDWLRQ WR HYHU\ERG\ ZH KDYH GHYHORSHG D QHZ V\V
WHP FDOOHG DanceReProducer WKDW FDQ DXWRPDWLFDOO\ JHQ

8VHUJHQHUDWHG YLGHR FOLSV FDOOHG MAD movies RU
mashup videos HDFK RI ZKLFK LV D GHULYDWLYH PL[WXUH RUFRPELQDWLRQ RI VRPH RULJLQDO YLGHR FOLSV DUH JDLQLQJ SRSXODULW\ RQ WKH ZHE DQG D ORW RI WKHP KDYH EHHQ XSORDGHG
DQG DUH DYDLODEOH RQ YLGHR VKDULQJ ZHE VHUYLFHV ,Q WKLV SDSHU ZH IRFXV RQ PXVLF YLGHR FOLSV RI GDQFH VFHQHV GDQFHYLGHR FOLSV LQ WKH IRUP RI 0$' PRYLHV RU PDVKXS YLGHRV6XFK D 0$' PXVLF YLGHR FOLS FRQVLVWV RI D PXVLFDO SLHFHDXGLR VLJQDOV DQG LPDJH VHTXHQFHV YLGHR IUDPHV WDNHQ
IURP RWKHU RULJLQDO YLGHR FOLSV 7KH RULJLQDO YLGHR FOLSVDUH FDOOHG 1st generation (primary or original) content DQG

an

reuse

reuse

1. INTRODUCTION

Copyright:

PictureMusic

Dance video

stricted use, distribution, and reproduction in any medium, provided the original
author and source are credited.
 KWWSHQZLNLSHGLDRUJZLNL0$' 0RYLH

3URFHHGLQJVRIWKH60&WK6RXQGDQG0XVLF&RPSXWLQJ&RQIHUHQFH-XO\3DGRYD,WDO\HUDWH D GDQFH YLGHR FOLS IRU DQ\ JLYHQ SLHFH RI PXVLF E\
VHJPHQWLQJ FRQFDWHQDWLQJ DQG VWUHWFKLQJ H[LVWLQJ GDQFHYLGHR FOLSV )LJXUH   7KLV V\VWHP SURYLGHV DQ LQWHUIDFH LQZKLFK D XVHU QRW RQO\ OLVWHQV WR PXVLF EXW DOVR HQMR\V PXVLF YLVXDOO\ E\ GLUHFWLQJ VXSHUYLVLQJ WKH VHPL DXWRPDWLF
JHQHUDWLRQ RI GDQFH YLGHR LPDJH VHTXHQFHV ,I WKH DXWRPDWLFDOO\ JHQHUDWHG YLGHR FOLS LV VDWLVIDFWRU\ WKH XVHU FDQ
MXVW ZDWFK LW EXW LI WKH XVHU GRHV QRW OLNH JHQHUDWHG LP
DJH VHTXHQFHV IRU VRPH PXVLFDO VHFWLRQV e.g. $ % DQG& LQ )LJXUH   WKH XVHU FDQ HDVLO\ FKRRVH DQRWKHU IDYRULWH
LPDJH VHTXHQFH IURP UDQNHG FDQGLGDWHV IRU HDFK PXVLFDOVHFWLRQ 7KHVH FDQGLGDWHV DUH DOVR DXWRPDWLFDOO\ SURSRVHG
E\ WKH V\VWHP DQG ZRXOG DOVR PDWFK D JLYHQ PXVLFDO VHFWLRQ RI WKH LQSXW SLHFH DFFRUGLQJ WR RXU PDSSLQJ PRGHO7KLV PDSSLQJ PRGHO ZDV WUDLQHG WKURXJK DQ DQDO\VLV RI D
ODUJH DPRXQW RI XVHUJHQHUDWHG GDQFH YLGHR FOLSV DYDLODEOH
RQ D YLGHR VKDULQJ ZHE VHUYLFH ,Q SDUWLFXODU ZH IRFXV RQ
WKH UHXVH RI YLGHR FOLSV RI WKH QG UG DQG N WK JHQHUDWLRQ
FRQWHQW )LJXUH  DV ZHOO DV WKH VW JHQHUDWLRQ FRQWHQW ,Q
RWKHU ZRUGV RXU V\VWHP HQDEOHV D XVHU WR JHQHUDWH D QHZPDVKXS YLGHR FOLS E\ UHXVLQJ H[LVWLQJ PDVKXS YLGHR FOLSVRQ WKH ZHEAutomatic mashup music video generation system

Web

Segment

Music video clips
Video1
Video2
Video3

...

...

...

...

VideoN
Stretch and Concatenate
OutputChorusImage sequence
Input

Music
Estimated music structure

A A B B B A A B B B C C C

Figure 2 $Q DXWRPDWLF PXVLF YLGHR JHQHUDWLRQ V\VWHP
DanceReProducer E\ UHXVLQJ H[LVWLQJ PXVLF YLGHR FOLSV• Impression 9LVXDO LPSUHVVLRQV VXFK DV GDQFH PR
WLRQ FRORU EULJKWQHVV DQG OLJKWLQJ DUH V\QFKURQL]HGZLWK WKH PXVLFDO LPSUHVVLRQContext relationships  FULWHULD IRU FRQWH[W V\QFKURQL]DWLRQ EHWZHHQ PXVLF DQG LPDJH VHTXHQFHV

2. RELATED WORK3UHYLRXV ZRUNV JHQHUDWHG YLVXDO SDWWHUQV EDVHG RQ VRPH
PXVLFDO DVSHFWV VXFK DV YLVXDOL]LQJ PXVLF FKRUGV E\FRORU >@ YLVXDOL]LQJ PXVLFDO PRRG >@ DQG FRQWUROOLQJD FRPSXWHUJUDSKLFV GDQFHU XQGHU PXVLFDO EHDWV > @
7KHUH ZHUH DOVR SUHYLRXV ZRUNV DXWRPDWLFDOO\ JHQHUDWLQJPXVLFV\QFKURQL]HG YLGHR E\ UHXVLQJ PHGLD FRQWHQW IRUH[DPSOH VRPH UHXVHG LPDJHV DQG SKRWRJUDSKV IURP WKH
ZHE > @ DQG RWKHUV UHXVHG KRPH YLGHRV > @ XQGHU DXGLR FKDQJHV >@ RU UHSHWLWLYH YLVXDO DQG DXUDO SDWWHUQV >@
3UHYLRXV ZRUNV KRZHYHU GLG QRW UHXVH GDQFH YLGHR FOLSVRQ WKH ZHE WR JHQHUDWH D QHZ PDVKXS YLGHR FOLS• Music structure 9LVXDO LPSUHVVLRQ WHPSRUDOFKDQJHV DUH V\QFKURQL]HG ZLWK WKH PXVLF VWUXFWXUH
e.g. YHUVH $ FKRUXV • Temporal continuity ,PDJH VHTXHQFH KDV WHPSRUDO FRQWLQXLW\ EXW YLVXDO LPSUHVVLRQ FDQ EH FKDQJHG
HDVLO\ RQ D PXVLF VWUXFWXUH ERXQGDU\
7KH DERYH FULWHULD DUH QRW DOO VDWLV¿HG DW DQ\ JLYHQ WLPHDQG DUH QRW PXWXDOO\ LQGHSHQGHQW +RZHYHU WKH\ SURYLGH
D XVHIXO IRXQGDWLRQ IRU JHQHUDWLQJ DQ LPDJH VHTXHQFH DS
SURSULDWH WR D SDUWLFXODU SLHFH RI PXVLF

3. SYSTEM DESIGN

3.2 Image sequence generation

7R GHYHORS 'DQFH5H3URGXFHU ZH ¿UVW FRQVLGHUHG WKH FULWHULD WKDW SHRSOH XVH LQ MXGJLQJ ³ZKDW LV DQ DSSURSULDWH LP
DJH VHTXHQFH IRU D SDUWLFXODU SLHFH RI PXVLF´ DV GHVFULEHG
EHORZ :H WKHQ GHVFULEH IXQFWLRQV RI WKH V\VWHP LQWHUIDFH

7KH PDVKXS YLGHR JHQHUDWLRQ GRQH PDQXDOO\ LV GLI¿FXOW
DQG WLPHFRQVXPLQJ 7R HQDEOH PRUH HI¿FLHQW JHQHUDWLRQRXU V\VWHP ¿UVW DXWRPDWLFDOO\ JHQHUDWHV DQ LPDJH VHTXHQFHDSSURSULDWH WR WKH PXVLF +RZHYHU WKH JHQHUDWHG VHTXHQFHPD\ QRW EH WR WKH XVHU¶V WDVWH ,Q VXFK FDVHV RWKHU VH
TXHQFH FDQGLGDWHV DUH VKRZQ RQ D VFUHHQ VR WKDW WKH XVHU
FDQ VLPSO\ FKRRVH D SUHIHUUHG RQH (YHQ WKRXJK LW ZRXOG
EH GLI¿FXOW IRU D XVHU WR PDQXDOO\ ¿QG DQRWKHU FDQGLGDWH
IURP DPRQJ D KXJH QXPEHU RI FDQGLGDWHV LW LV HDV\ WR LQ
WHUDFWLYHO\ FKRRVH D SUHIHUUHG FDQGLGDWH
:H SURYLGH DQ RYHUYLHZ RI WKH LQWHUIDFH¶V LPDJH VH
TXHQFH JHQHUDWLRQ DQG IXQFWLRQV EHORZ

3.1 Criteria of natural/skillful relationships between
an image sequence and music
7R GHVLJQ WKH V\VWHP ZH FRQVLGHUHG WKH FULWHULD IURPWZR DVSHFWV ± ORFDO UHODWLRQVKLSV DQG FRQWH[W JOREDO UHODWLRQVKLSV H[SODLQHG EHORZ ± WDNLQJ LQWR DFFRXQW SUHYLRXVZRUN > @ DQG WKH FRPPHQWV RIIHUHG E\ KXPDQ FUHDWRUV
RI 0$' PRYLHV 

3.2.1 Automatic image sequence generation

Local relationships  FULWHULD IRU LPSUHVVLRQ V\QFKURQL]DWLRQ EHWZHHQ WKH PXVLF DQG LPDJH VHTXHQFHV

7R UHXVH H[LVWLQJ FRQWHQW ZH ¿UVW JDWKHU GDQFH YLGHR FOLSVRQ D YLGHR VKDULQJ ZHE VHUYLFH DQG WKH V\VWHP HVWLPDWHV
WKH WHPSR DQG EDU OLQH RI WKH PXVLF DXGLR VLJQDOV LQ WKRVHYLGHR FOLSV :H DVVXPH WKH PXVLF DQG LWV GDQFH PRWLRQVZLWKLQ HDFK YLGHR FOLS DUH V\QFKURQL]HG ZKLOH GHDOLQJ ZLWKWKH ORFDO UHODWLRQVKLSV DQG XVH HDFK EDU PHDVXUH RI WKH• Rhythm 9LVXDO UK\WKPV VXFK DV GDQFH PRWLRQ FDPHUD ZRUN DQG FXW e.g. GLVVROYH DUH V\QFKURQL]HGZLWK EHDW DQG PXVLFDO DFFHQW
 6RPH FUHDWRUV GLVFORVHG WKHLU FUHDWLYH SURFHVVHV RQ WKH ZHE

3URFHHGLQJVRIWKH60&WK6RXQGDQG0XVLF&RPSXWLQJ&RQIHUHQFH-XO\3DGRYD,WDO\

1
2
8
7
6
3
5
4
Figure 4 ([DPSOH RI LQWHUDFWLYH VHTXHQFH VHOHFWLRQ )RXU
GLIIHUHQW LPDJH VHTXHQFH FDQGLGDWHV DUH SUHYLHZHG DQG WKH
ORZHUULJKW FDQGLGDWH LV FKRVHQ E\ D XVHU

Figure 3 ([DPSOH RI WKH 'DQFH5H3URGXFHU VFUHHQPXVLF DV WKH PLQLPXP XQLW IRU VHJPHQWLQJ DQG FRQFDWHQDWLQJ LPDJH VHTXHQFHV +HUHDIWHU ZH GHQRWH DQ LPDJH
VHTXHQFH VHULHV RI YLGHR IUDPHV IRU WKH EDUOHYHO PLQLPXP XQLW DV D visual unit
6HFRQG WKH V\VWHP VHDUFKHV IRU D YLVXDO XQLW DSSURSULDWHWR HDFK EDU IRU WKH LQSXW WDUJHW PXVLFDO SLHFH 7KH XQLWV
DUH WLPHVWUHWFKHG XQGHU WKH WHPSR RI WKH LQSXW PXVLF
DQG WKHQ DUH FRQFDWHQDWHG WR JHQHUDWH DQ LPDJH VHTXHQFH
,Q WKLV UHJDUG WR GHDO ZLWK WKH FRQWH[W UHODWLRQVKLSV WKH
V\VWHP VHOHFWV YLVXDO XQLWV ZKLFK WDNH LQWR DFFRXQW PXVLF
VWUXFWXUH DQG WHPSRUDO FRQWLQXLW\
7R VDWLVI\ HDFK FULWHULRQ GHVFULEHG LQ  ZH LPSOHPHQW
WKH IROORZLQJ SURFHVVHV7KLV LQWHUIDFH DOVR SURYLGHV WKH IROORZLQJ IXQFWLRQV WR UH
ÀHFW WKH XVHU¶V SUHIHUHQFHVInteractive re-selection of a generated image sequence%\ FOLFNLQJ WKH 1* EXWWRQ )LJXUH  
  WKH XVHU
FDQ VHH RWKHU VHTXHQFH FDQGLGDWHV RQ D VFUHHQ DQG
VLPSO\ FKRRVH WKH SUHIHUUHG RQH )LJXUH   
7KH XVHU FDQ VHH DQG FRPSDUH GLIIHUHQW FDQGLGDWHVGXULQJ SOD\EDFN DQG FDQ FKRRVH KLVKHU IDYRULWH
VHTXHQFH 6LQFH WKLV LQWHUDFWLYH UHVHOHFWLRQ IXQF
WLRQ ZRUNV RQ HDFK VHFWLRQ RI WKH PXVLF VWUXFWXUH
e.g. $ % DQG & LQ )LJXUH   WKH XVHU FDQ XVH WKLVIXQFWLRQ WR HDVLO\ FRQVLGHU WKH PXVLF VWUXFWXUH DQG
FRQWH[W
Jumping to the beginning of sections %\ FOLFNLQJ WKHMXPS EXWWRQ )LJXUH  
 RU YLVXDOL]HG VHFWLRQV  D XVHU FDQ GLUHFWO\ MXPS WR DQG YLHZ WKH
SUHYLRXV RU WKH QH[W VHFWLRQ RI D VRQJ

Rhythmic synchronization $ PXVLFDO EDU LV XVHG DV WKH
PLQLPXP XQLW IRU VHJPHQWLQJ DQG FRQFDWHQDWLQJ $YLVXDO XQLW LV VWUHWFKHG XQGHU LQSXW PXVLF WHPSR
Impression synchronization %\ PRGHOLQJ WKH PDSSLQJ
EHWZHHQ WKH H[WUDFWHG DXGLR DQG YLVXDO IHDWXUHV IRU
LPSUHVVLRQ WKH V\VWHP DXWRPDWLFDOO\ VHOHFWV DQ DSSURSULDWH YLVXDO XQLW WR LQSXW PXVLF LPSUHVVLRQ LQHDFK EDUMusic structure DQG Temporal continuity %\ LQWURGXFLQJFRVWV UHSUHVHQWLQJ WKH WHPSRUDO FRQWLQXLW\ DQG PXVLF
VWUXFWXUH RI WKH JHQHUDWHG VHTXHQFH WKH V\VWHP DXWRPDWLFDOO\ VHOHFWV DQ LPDJH VHTXHQFH FRQVLGHULQJ WKH
FRQWH[W UHODWLRQVKLSV4. INTERNAL MECHANISM OFDANCEREPRODUCER
7R GHYHORS 'DQFH5H3URGXFHU ZH PRGHOHG WKH UHODWLRQVKLSV EHWZHHQ PXVLF DQG YLGHR DQG WKHQ JHQHUDWHG LPDJH
VHTXHQFHV DSSURSULDWH WR LQSXW PXVLF E\ FRQVLGHULQJ WKH
ORFDO DQG FRQWH[W UHODWLRQVKLSV ,Q JHQHUDO LW LV GLI¿FXOWWR PRGHO VXFK UHODWLRQVKLSV EXW ZH VROYHG WKLV SUREOHPWKURXJK WUDLQLQJ XVLQJ D KXJH TXDQWLW\ PDVKXS YLGHR FOLSVSRVWHG WR WKH ZHE 6LQFH WKH FRQWHQW YLGHRV ZHUH PDGH E\
KXPDQV WKHUH ZHUH YDULRXV W\SHV RI PXWXDO UHODWLRQVKLSEHWZHHQ WKH PXVLF DQG WKH LPDJH VHTXHQFHV 7KLV VXJJHVWV WKDW VXFK YLGHRV FDQ EH XVHG WR OHDUQ WKH UHODWLRQVKLSVWKURXJK D PDFKLQHOHDUQLQJ WHFKQLTXH
0RGHOLQJ XVLQJ WKH PDVKXS FOLSV VXIIHUV IURP WZR SURE
OHPV 2QH LV WKDW FRPSOH[ UHODWLRQVKLSV H[LVW VXFK DVZKHUH ³WKH VDPH LPDJH VHTXHQFHV DUH XVHG IRU GLIIHUHQWPXVLF´ RU ³GLIIHUHQW LPDJH VHTXHQFHV DUH XVHG IRU WKH VDPH
PXVLF´ )LJXUH   $QRWKHU SUREOHP LV WKDW WKH YLGHR TXDOLW\ YDULHV VWURQJO\ DQG LW LV GLI¿FXOW WR MXGJH WKH SRVVLELOLW\

3.2.2 Interface
6FUHHQVKRWV RI WKH LPSOHPHQWHG 'DQFH5H3URGXFHU LQWHUIDFH DUH VKRZQ LQ )LJXUH  DQG  7KHUH DUH EDVLF IXQFWLRQVIRU YLHZLQJ VXFK DV D ZLQGRZ VKRZLQJ WKH JHQHUDWHG LP
DJH VHTXHQFH )LJXUH    IXQFWLRQV WR ORDG LQSXW PXVLF
DQG VDYH WKH JHQHUDWHG YLGHR   WR SOD\ DQG VWRSSDXVH
WKH JHQHUDWHG YLGHR   DQG D SOD\EDFNSRVLWLRQ ³VOLGHU´
DQG WKH PXVLF VWUXFWXUH HVWLPDWHG DXWRPDWLFDOO\ >@  
7KH JUHHQ UHFWDQJXODU PDUNHUV LQ WKH PXVLF VWUXFWXUH UHSUH
VHQW FKRUXV VHFWLRQV DQG WKH EOXH PDUNHUV UHSUHVHQW RWKHU
VHFWLRQV ,Q DGGLWLRQ WKH WRWDO GXUDWLRQ RI WKH LQSXW PXVLF
LV HTXDOO\ GLYLGHG LQWR  VHFWLRQV 
 



3URFHHGLQJVRIWKH60&WK6RXQGDQG0XVLF&RPSXWLQJ&RQIHUHQFH-XO\3DGRYD,WDO\

Database construction
A Gather videos B Extract frame feature C Extract bar-level feature E Construct
Resampling
Database...
1 frameImage sequenceWebmusic video clips(30 fps, 44.1kHz)...

Dance video

1 bar

30 fpsResampling

View countView count(16 points)

D

Music

...

44.1kHz

DCT

1 bar

Beat tracking

1 feature vector ( =...

(3rd order with DC)+)
View countVideo generation
Input music

G Reconstruct H Train mapping model
Database
Feature space

User
...

Music

F
Extract bar-level feature
0%

...

input
tempo

PCA

> 20%

g
htin
weig lation
u
lc
a
c

DCT

View countOutputViterbi searchunder the Euclidean distance...

Visual...

clustering

<2

...

mapping

I Select visual unit...

linearregression

...
A A A

A

Music
B B structure

Stretch and concatenateof the unit...

Figure 5 2YHUYLHZ RI 'DQFH5H3URGXFHU D GDQFH YLGHR DXWKRULQJ V\VWHP WKDW FDQ DXWRPDWLFDOO\ JHQHUDWH D GDQFH YLGHRFOLS DSSURSULDWH WR D JLYHQ SLFH RI PXVLF E\ VHJPHQWLQJ FRQFDWHQDWLQJ DQG VWUHWFKLQJ H[LVWLQJ GDQFH YLGHR FOLSV7KH V\VWHP ¿UVW FDOFXODWHV WKH SRZHU RI WKH LQSXW DXGLR
VLJQDO DQG WKHQ FDOFXODWHV LWV DXWRFRUUHODWLRQ YDOXHV DQG
HVWLPDWHV WKHLU SHDN WLPH 6LQFH LW UHSUHVHQWV WKH SHULRGLF
LW\ RI WKH SRZHU ZH XVH WKH WLPH DV WHPSR RQH EHDW WLPH ,Q WKLV UHJDUG WR DYRLG RFWDYH HUURU e.g. KDOIGRXEOHWHPSR HUURU  WKH HVWLPDWLRQ LV OLPLWHG WR WHPSR ZLWKLQ D
UDQJH RI 60 − 120 ESP EHDW SHU PLQXWH 
6HFRQG WKH V\VWHP FDOFXODWHV FURVVFRUUHODWLRQ EHWZHHQ
WKH SRZHU DQG WKH SXOVH VLJQDO JHQHUDWHG XQGHU WKH HVWL
PDWHG WHPSR 6LQFH WKH SHDN WLPH RI WKH FURVVFRUUHODWLRQUHSUHVHQWV WKH ¿UVW EHDW WLPH WKH V\VWHP UHJDUGV WKH WLPH
DV WKH EHJLQQLQJ WLPH RI WKH ¿UVW EDU ,Q DGGLWLRQ ZH DV
VXPH WKDW WKH GDWDVHW YLGHRV KDYH D OHQJWK RI  EHDWV RQH
PHDVXUH LQ  WLPH  DQG WKHQ WKH V\VWHP GHFLGHV DOO EDUOLQHV PHFKDQLFDOO\RI LWV UHXVH 7KHVH REVWDFOHV PDNH LW GLI¿FXOW WR PRGHO WKHUHODWLRQVKLSV DQG ZHUH QRW GHDOW ZLWK SUHYLRXV ZRUNV)LJXUH  JLYHV DQ RYHUYLHZ RI WKH 'DQFH5H3URGXFHU V\V
WHP 7KH V\VWHP FRQVLVWV RI WZR SURFHGXUHV GDWDEDVH FRQVWUXFWLRQ DQG YLGHR JHQHUDWLRQ ,Q WKLV VHFWLRQ ZH GHVFULEH
WKH GHWDLOV RI WKH V\VWHP DQG H[SODLQ KRZ ZH VROYH WKH
DERYH WZR SUREOHPV LQ PRGHOLQJ XVLQJ WKH PDVKXS FOLSV
4.1 Database construction,Q WKH GDWDEDVH FRQVWUXFWLRQ GDWDEDVH YLGHRV DUH JDWKHUHG
YLD WKH ZHE DQG WKHQ DXGLR DQG YLVXDO IHDWXUHV DUH H[WUDFWHG
IURP WKH YLGHRV WKURXJK WKH IROORZLQJ VWHSV6WHS  *DWKHU GDQFH PXVLF YLGHRV YLD ZHE DQG UHVDPSOH
WKH VDPSOLQJ IUHTXHQF\ RI WKH PXVLF WR  N+]DQG WKH IUDPHUDWH RI WKH LPDJH VHTXHQFH WR  ISV)LJXUH  
$ 
6WHS  (VWLPDWH EDU OLQH RI WKH YLGHRV E\ XVLQJ EHDW WUDFN
LQJ WHFKQLTXHV % 
6WHS  ([WUDFW IHDWXUH YHFWRUV WR OHDUQ WKHLU UHODWLRQVKLS±% &  6LQFH WKH DQDO\VLV IUDPH PDWFKHV WKH IUDPH
UDWH WKH GLVFUHWH WLPH VWHS  frame-time LV DERXW PV DERXW  SRLQWV  7KH H[WUDFWHG IHDWXUHVLQ HDFK IUDPHWLPH DUH FDOOHG frame features 7KH
IUDPH IHDWXUHV DUH WKHQ LQWHJUDWHG LQ HDFK EDU WR RE
WDLQ ZKDW DUH FDOOHG bar-level features

4.1.2 Frame feature extraction (Music)
7KH IUDPH IHDWXUHV RI PXVLF DUH GH¿QHG ZLWK WKH KHOS RI
SUHYLRXV ZRUN RQ UHODWLRQVKLSV EHWZHHQ DXGLR DQG YLVXDO> @ DQG PXVLFDO JHQUH FODVVL¿FDWLRQ >@ 7KHVH IHDWXUHV UHSUHVHQW PXVLFDO DFFHQWV DQG LPSUHVVLRQV$V WKH IUDPH IHDWXUHV IRU DFFHQWV WR UHSUHVHQW WHPSRUDOFKDQJH LQ WKH SRZHU RI WKH DXGLR VLJQDO ZH H[WUDFW WKH ¿O
WHU EDQN RXWSXW 4 GLPV DQG VSHFWUDO ÀX[ 1 GLP  $V WKH
IUDPH IHDWXUHV IRU LPSUHVVLRQV WR UHSUHVHQW WLPEUH ZH H[WUDFW WKH ]HURFURVVLQJ UDWH 1 GLP DQG WK RUGHU 0)&&V
PHOIUHTXHQF\ FHSVWUDO FRHI¿FLHQWV ZLWK D '& FRPSRQHQW 13 GLPV 

4.1.1 Beat tracking

4.1.3 Frame feature extraction (Image sequence)

0XFK ZRUN KDV EHHQ GRQH RQ EHDW WUDFNLQJ >@ DQG ZH
SODQ WR IRFXV RQ XVLQJ VXFK WHFKQLTXHV LQ WKH IXWXUH EXW RXU
FXUUHQW LPSOHPHQWDWLRQ LV D VLPSOH RQH ZKLFK ZDV HIIHFWLYH
LQ RXU SUHOLPLQDU\ H[SHULPHQW

7KH IUDPH IHDWXUHV RI DQ LPDJH VHTXHQFH DUH GH¿QHG ZLWKWKH KHOS RI SUHYLRXV ZRUN RQ UHODWLRQVKLSV EHWZHHQ DXGLRDQG YLVXDO >@ 7KHVH IHDWXUHV UHSUHVHQW YLVXDO DFFHQWV

3URFHHGLQJVRIWKH60&WK6RXQGDQG0XVLF&RPSXWLQJ&RQIHUHQFH-XO\3DGRYD,WDO\DQG LPSUHVVLRQV 7R H[WUDFW WKH IHDWXUHV WKH LPDJH UHVROXWLRQ LV UHVDPSOHG WR 128 × 96
$V WKH IUDPH IHDWXUHV IRU DFFHQWV WR UHSUHVHQW FDPHUDZRUN DQG GDQFH PRWLRQ DQG UHODWHG WHPSRUDO FKDQJHV ZH
H[WUDFW WKH PHDQ YDOXHV RI WKH WHPSRUDO GHULYDWLYH RI WKHZHOONQRZQ RSWLFDO ÀRZ DQG EULJKWQHVV 2 GLPV  :H XVH
D EORFNPDWFKLQJ DOJRULWKP WR GHWHFW WKH RSWLFDO ÀRZ IURPLPDJH VHTXHQFHV ZH XVH D 64 × 48 EORFN ZKLFK LV VKLIWHG
E\  PD[LPXP UDQJH LV   7KH IUDPH IHDWXUHV IRU LP
SUHVVLRQV DUH WKH PHDQ YDOXHV DQG VWDQGDUG GHYLDWLRQV RIWKH KXH VDWXUDWLRQ DQG EULJKWQHVV YDOXHV 6 GLPV  ,Q DG
GLWLRQ GLPHQVLRQDO '&7 GLVFUHWH FRVLQH WUDQVIRUP FRHI¿FLHQWV DUH H[WUDFWHG 4 GLPV IRU YHUWLFDO DQG 3 GLPV IRU
KRUL]RQWDO 

4.2.1 Linear regression models for multiple clusters,Q WKLV SDSHU D ORFDO FRVW LV FDOFXODWHG E\ D OLQHDU UHJUHVVLRQ PRGHO ZKLFK LV XVHG WR OHDUQ WKH UHODWLRQVKLSV EH
WZHHQ WKH DXGLR DQG YLVXDO EDUOHYHO IHDWXUHV +RZHYHUWR PRGHO FRPSOH[ UHODWLRQVKLSV VXFK DV ³WKH VDPH YLVXDOXQLWV DUH XVHG IRU GLIIHUHQW PXVLF´ RU ³GLIIHUHQW YLVXDO XQLWVDUH XVHG IRU WKH VDPH PXVLF´ )LJXUH   RQH UHJUHVVLRQPRGHO LV LQVXI¿FLHQW
7KHUHIRUH ZH SURSRVH D OLQHDU UHJUHVVLRQ ZKHUH WKH V\V
WHP XVHV OLQHDU UHJUHVVLRQ PRGHOV IRU PXOWLSOH FOXVWHUV
7KH PXOWLSOH FOXVWHUV DUH REWDLQHG E\ DSSO\LQJ kPHDQV
FOXVWHULQJ WR IHDWXUH YHFWRUV ZKHUH D IHDWXUH YHFWRU LV GH
¿QHG DV D FRQFDWHQDWLRQ RI D EDUOHYHO DXGLR IHDWXUH RIPXVLF DQG D EDUOHYHO YLVXDO IHDWXUH RI LPDJH VHTXHQFHV
LQ WKH GDWDEDVH 1RWH WKDW WKLV IHDWXUH YHFWRU LV XVHG MXVW IRUWKH FOXVWHULQJ )RU HDFK FOXVWHU D OLQHDU UHJUHVVLRQ PRGHOLV WUDLQHG VR WKDW EDUOHYHO YLVXDO IHDWXUHV FDQ EH SUHGLFWHG
E\ EDUOHYHO DXGLR PXVLF IHDWXUHV )LJXUH  
+ 

4.1.4 Bar-level feature extraction:H SURSRVH D bar-level feature ZKLFK LV DQ LQWHJUDWLRQ RI
WKH IUDPH IHDWXUHV LQ HDFK EDU 7R H[WUDFW IHDWXUHV IURP RQH
SLHFH RI PXVLF RU RQH YLGHR FOLS LQ PRVW SUHYLRXV ZRUN
e.g. PXVLFDO JHQUH FODVVL¿FDWLRQ LQWHJUDWLRQ ZDV GRQH XVLQJ WKH WLPH DYHUDJH DQG LWV VWDQGDUG GHYLDWLRQ >@ +RZHYHU VXFK LQWHJUDWLRQ GURSV WHPSRUDO LQIRUPDWLRQ RI WKHDXGLRYLVXDO IHDWXUHV,Q WKLV SDSHU ZH LQWHJUDWH WKHVH IUDPH IHDWXUHV WR EDUOHYHO IHDWXUHV YLD XVLQJ '&7 )LJXUH  
'  ,Q HDFK EDUIUDPH IHDWXUHV DUH UHVDPSOHG WR  SRLQWV IRU WKH WLPH D[LVWKH V\VWHP FRPSXWHV '&7 IRU HDFK GLPHQVLRQ DQG WKHQ
WKH UG RUGHU '&7 FRHI¿FLHQWV ZLWK D '& FRPSRQHQW XVHG
DV WKH EDUOHYHO IHDWXUHV 7KHUHIRUH WKH QXPEHU RI GLPHQVLRQV RI WKH EDUOHYHO IHDWXUHV LV IRXU WLPHV WKH QXPEHU
IURP WKH IUDPH IHDWXUHV

4.2.2 Image sequence selection under the criteria fornatural/skillful relationships%\ LQWURGXFLQJ FRVWV UHSUHVHQWLQJ WKH ORFDO DQG FRQWH[W UHODWLRQVKLSV ZH FDQ VROYH WKLV YLGHR JHQHUDWLRQ SUREOHP E\
PLQLPL]LQJ WKH FRVWV WKURXJK D 9LWHUEL VHDUFK )LJXUH ,  7KH PRGHO RI WKH FOXVWHU KDYLQJ WKH FHQWURLG QHDUHVWWR WKH LQSXW IHDWXUHV LV VHOHFWHG DQG YLVXDO IHDWXUHV DSSURSULDWH WR WKH LQSXW DXGLR IHDWXUHV DUH HVWLPDWHG E\ XVLQJ
WKH PRGHO 7R FDOFXODWH WKH FRVWV RI WKH ORFDO UHODWLRQVKLSVWKH V\VWHP FDOFXODWHV WKH GLVWDQFH EHWZHHQ WKH HVWLPDWHGIHDWXUHV DQG WKH YLVXDO IHDWXUHV RI DOO XQLWV
7R UHSUHVHQW WKH FRVWV RI WKH FRQWH[W UHODWLRQVKLSV D PX
VLFDO VWUXFWXUH DQG FKRUXV VHFWLRQ DUH HVWLPDWHG XVLQJ 5H
IUDL' >@ 7KH HVWLPDWHG EHJLQQLQJ DQG HQGLQJ WLPHV RI DOO
VHFWLRQV DUH XVHG DV WKH ERXQGDULHV RI D PXVLFDO VHFWLRQ+RZHYHU VHFWLRQV OHVV WKDQ  EDUV LQ OHQJWK DUH QRW XVHG DV
D VHFWLRQ IRU WKLV SXUSRVH
/HW d(n, km ) EH WKH (XFOLGHDQ GLVWDQFH UHSUHVHQWLQJ WKH
ORFDO FRVW EHWZHHQ WKH n(1 ≤ n ≤ N )WK EDU OHYHO IHDWXUHRI WKH LQSXW DQG WKH mWK YLGHR¶V kWK XQLW¶V IHDWXUHV RI WKH
GDWDEDVH 7KH FDOFXODWHG ORFDO FRVWV DQG DFFXPXODWHG FRVWVDUH GH¿QHG DV IROORZV⎧⎪LI ch(n) = 1⎨d(n, km )cl (n, km ) =∧ch(km ) = 1 ,⎪⎩pc × d(n, km ) RWKHUZLVH

4.2 Video generation,Q WKH YLGHR JHQHUDWLRQ WR VHOHFW YLVXDO XQLWV IRU HDFKIUDPH IURP WKH GDWDEDVH WKH V\VWHP SURFHVV FRQVLVWV RI
WKH IROORZLQJ VWHSV6WHS  ([WUDFW WKH EDUOHYHO IHDWXUHV RI D JLYHQ PXVLFDO
SLHFH )LJXUH  
) 
6WHS  5HFRQVWUXFW WKH GDWDEDVH *  7R DYRLG JHQHUDWLQJ D YLGHR ZLWK XQQDWXUDOO\ IDVWVORZ WHPSR YL
VXDO XQLWV ZLWK WHPSL 20% DERYH RU EHORZ WKH LQSXWWHPSR DUH QRW XVHG IRU WKH IROORZLQJ VWHSV6WHS  $SSO\ 3&$ SULQFLSDO FRPSRQHQW DQDO\VLV IRUDOO EDUOHYHO IHDWXUHV RI DOO EDUV DQG VWRUH ORZ N 
GLPHQVLRQDO IHDWXUHV 7KH N GLPHQVLRQ LV GHFLGHG
EDVHG RQ WKH FXPXODWLYH FRQWULEXWLRQ UDWLR ≤ 95% )RU RXU LQYHVWLJDWLRQV WKH GLPHQVLRQV RI DXGLR DQG
YLVXDO IHDWXUHV GHVFULEHG DERYH ZHUH UHGXFHG IURP
76 WR 62 DQG IURP 80 WR 68 UHVSHFWLYHO\ 
6WHS  0RGHO UHODWLRQVKLS EHWZHHQ PXVLF DQG LPDJH VH
TXHQFH IURP WKH GDWDEDVH +  7KLV VWHS LV H[SODLQHG LQ PRUH GHWDLO EHORZ VHFWLRQ 4.2.1 
6WHS  6HOHFW YLVXDO XQLWV XQGHU WKH FULWHULD RI WKH UHODWLRQVKLSV GHVFULEHG LQ  , 

ca (n, km )⎧cl (n, km )LI (μ = m ∧ κ = k − 1)⎪⎪⎪
⎨ +c (n − 1, κ ) ∨st(n) = st(n − 1)
a
μ
.
= min
τ,μ
⎪
pt × cl (n, km )⎪⎪⎩+ca (n − 1, κμ ) RWKHUZLVH

ZKHUH ch(n) UHWXUQV  LI n LV LQFOXGHG LQ D FKRUXV VHFWLRQ
DQG st(n) UHWXUQV WKH QXPEHU RI PXVLFDO VHFWLRQV $ KLJKHU
pc YDOXH PHDQV WKDW WKH XQLW RI FKRUXV VHFWLRQV DUH PRUHHDVLO\ VHOHFWHG DW D FKRUXV VHFWLRQ $ ORZHU pt YDOXH PHDQVWKDW WKH VHOHFWHG XQLW KDV OHVV WLPH FRQWLQXLW\ 7R PLQL
PL]H WKH DFFXPXODWHG FRVW DW WKH N PHDVXUH WKH V\VWHP

 6LQFH WKH GDWDEDVH LV UHFRQVWUXFWHG GHSHQGLQJ RQ WKH WHPSR RI WKH
LQSXW WKH UHGXFHG GLPHQVLRQ LV QRW FRQVWDQW3URFHHGLQJVRIWKH60&WK6RXQGDQG0XVLF&RPSXWLQJ&RQIHUHQFH-XO\3DGRYD,WDO\&RQGLWLRQ  (DFK YLGHR FOLS KDV WKH YLHZ FRXQW E\ XVHUVRQ WKH ZHE&RQGLWLRQ  7KH QXPEHU RI DYDLODEOH YLGHR FOLSV LV ODUJHHQRXJKVHOHFWV D XQLW ZKLFK KDV PLQLPXP DFFXPXODWHG FRVW dmin 
DQG WKHQ D LPDJH VHTXHQFH LV JHQHUDWHG E\ EDFNWUDFLQJ
dmin = argmink,mca (N, km ).



$V FRQWHQW IXO¿OOLQJ DOO RI WKH DERYH FRQGLWLRQV ZH
XVHG PDVKXS YLGHRV ZKLFK DUH JHQHUDWHG IURP -DSDQHVH
GDQFH VLPXODWLRQ JDPHV IXOO RI GDQFH VFHQHV ³7+(,'2/0#67(5´ DQG ³7+( ,'2/0#67(5 /,9( )25<28´  ,Q DGGLWLRQ ZH DOVR XVHG GDQFH YLGHRV ZKLFKDUH JHQHUDWHG XVLQJ MikuMikuDance (MMD) WKDW LV D 
GLPHQWLRQDO KXPDQ PRWLRQ V\QWKHVL]HU IRU GDQFH SHUIRUPDQFH %RWK YLGHRV FDQ EH IRXQG RQ D YLGHR VKDULQJ VHUYLFH NicoNicoDouga  7R FRQVWUXFW D GDWDEDVH ZH JDWKHUHG  RI WKHVH PDVKXS YLGHR FOLSV DQG  RI WKHVH
00' YLGHR FOLSV DOO RI ZKLFK KDG WKH YLHZ FRXQW RI RYHU
 RQ WKH 1LFR1LFR'RXJD

7KH LQWHUDFWLYH UHVHOHFWLRQ IXQFWLRQ LV LPSOHPHQWHG VR
WKDW WKH V\VWHP FKRRVHV IRXU GLIIHUHQW FDQGLGDWHV IRU HDFKVHFWLRQ )LJXUH   7KHVH FDQGLGDWHV DUH PDGH IURP
IRXU GLIIHUHQW DFFXPXODWHG FRVWV DQG WKHQ IRXU LPDJH VH
TXHQFHV DUH JHQHUDWHG E\ EDFNWUDFLQJ 7R H[SDQG WKH YD
ULHW\ RI JHQHUDWHG LPDJH VHTXHQFHV WKH FKRVHQ FDQGLGDWHV
DUH PDGH IURP PLQLPXP  PLQLPXP  PLQLPXP DQG
PD[LPXP DFFXPXODWHG FRVWV 7KLV HQDEOHV JHQHUDWLRQ RI D
YDULDWLRQDO LPDJH VHTXHQFH
4.3 Model training weighted according to view counts
7KLV SDSHU IRFXVHV RQ WKH UHXVH RI WKH 0$' PRYLHV DYDLO
DEOH RQ WKH ZHE 6LQFH WKHUH DUH PDQ\ FUHDWRUV WKH DXWKRULQJ TXDOLW\ RI JHQHUDWHG YLGHRV YDULHV ZLGHO\ ,Q RWKHUZRUGV HDFK YLGHR ZLOO KDYH D GLIIHUHQW OHYHO RI UHOLDELOLW\ UHJDUGLQJ WKH UHODWLRQVKLSV EHWZHHQ PXVLF DQG LPDJH
:H DVVXPH WKDW D YLGHR JHQHUDWHG E\ D XVHU KDYLQJ JRRG
0$' PRYLH VNLOOV ZLOO KDYH KLJKHU UHOLDELOLW\ DQG KLJKHUSRVVLELOLW\ RI LWV UHXVH 7KHUHIRUH WR PRGHO DQ DSSURSULDWH LPDJH VHTXHQFH WR SDUWLFXODU PXVLF WKH V\VWHP VKRXOGLQWURGXFH D ZHLJKWLQJ IDFWRU LQ WKH PRGHO WUDLQLQJ SURFHVVZKHUH KLJKHU TXDOLW\ YLGHR ZLOO EH JLYHQ D JUHDWHU ZHLJKW
7R HQDEOH DXWRPDWLF MXGJPHQW RI WKH TXDOLW\ ZH LQWURGXFH WKH LGHD RI XVLQJ WKH YLHZ FRXQW RI HDFK YLGHR FOLSRQ WKH ZHE DV D ZHLJKW VLQFH WKH YLHZ FRXQW UHÀHFWV WKH
YLGHR TXDOLW\ /HW ω EH DQ LQWHJHU ZHLJKWLQJ IDFWRU GH¿QHG
DV IROORZV ZKHUH Vc LQGLFDWHV WKH YLHZ FRXQW
w=

max (α × log10 (Vc ) + 0.5 + β, 0) .

5.2 Trial usage and introspective comments
0DQ\ YLGHRV JHQHUDWHG E\ 'DQFH5H3URGXFHU ZHUH V\QFKURQL]HG UHJDUGLQJ UK\WKP DQG LPSUHVVLRQ EHWZHHQ WKH
PXVLF DQG LPDJH VHTXHQFH 7KLV VXJJHVWV WKDW WKH V\VWHP
FDQ EH HIIHFWLYH DQG WKH PRGHOLQJ LV DSSURSULDWH
7ULDO XVHUV RI WKH V\VWHP RIIHUHG FRPPHQWV HVSHFLDOO\UHJDUGLQJ WKH HIIHFWLYHQHVV RI WKH LQWHUDFWLYH UHVHOHFWLRQIXQFWLRQ $ W\SLFDO FRPPHQW ZDV WKDW ³WKH IXQFWLRQ ZDVXVHIXO DQG HIIHFWLYH´ KRZHYHU LQ FRQWUDVW DQRWKHU XVHUFRPPHQWHG WKDW ³RFFDVLRQDOO\ WKHUH ZDV QR DSSURSULDWH
FDQGLGDWH´
6RPH FRPPHQWV ZHUH RQ ZD\V WR LPSURYH WKH V\VWHPSHUIRUPDQFH 2QH XVHU ZKR KDG QR H[SHULHQFH LQ 0$'
PRYLH JHQHUDWLRQ VDLG LW ZRXOG EH XVHIXO WR KDYH ³PRUH
FDQGLGDWHV IRU WKH LPDJH VHTXHQFH´ $QRWKHU FRPPHQWIURP D XVHU ZKR KDG 0$' PRYLH H[SHULHQFH ZDV WKDWWKH V\VWHP QHHGHG DQ ³DGMXVWPHQW IXQFWLRQ IRU WKH EDU DQG
ERXQGDU\ RI WKH PXVLFDO VHFWLRQ ´,Q RXU FXUUHQW LPSOHPHQWDWLRQ α DQG β DUH VHW WR 2 DQG−7 UHVSHFWLYHO\ 7KLV PHDQV D YLHZ FRXQW RI 10, 000 FRUUHVSRQGV WR ω = 1 ZKLOH D YLHZ FRXQW RI 100, 000 FRUUHVSRQGV WR ω = 3 7R LPSOHPHQW WKH ZHLJKWHG WUDLQLQJ WKH
QXPEHU RI EDUOHYHO DXGLRYLVXDO IHDWXUHV WUDLQLQJ VDPSOHV RI D YLGHR FOLS LV YLUWXDOO\ LQFUHDVHG E\ LWV ω GRXEOHG E\ ω = 2 IRU H[DPSOH LQ WUDLQLQJ OLQHDU UHJUHVVLRQPRGHOV

6. CONCLUSION
DanceReProducer LV D GDQFH YLGHR DXWKRULQJ V\VWHP WKDWFDQ DXWRPDWLFDOO\ JHQHUDWH GDQFH YLGHR DSSURSULDWH WR PXVLF E\ UHXVLQJ H[LVWLQJ GDQFH YLGHR VHTXHQFHV 7ULDO XVDJH
RI WKH V\VWHP KDV VKRZQ WKDW LW LV D XVHIXO WRRO IRU XVHUVZLWK OLWWOH NQRZOHGJH RU H[SHULHQFH LQ 0$' PRYLH JHQHUDWLRQ  $OWKRXJK GDQFH YLGHR FRQWHQW LV FXUUHQWO\ VXSSRUWHG LQ RXU LPSOHPHQWDWLRQ RXU DSSURDFK KDV FDSDELOLW\WR XWLOL]H IRU DQ\ RWKHU PXVLF YLGHR FOLSV2QH EHQH¿W RI 'DQFH5H3URGXFHU LV WKDW D XVHU GRHVQRW QHHG WR HQJDJH LQ WLPHFRQVXPLQJ PDQXDO JHQHUDWLRQ0RUHRYHU WKH ³UHXVH´ DSSURDFK GHVFULEHG LQ WKLV SDSHU
LV QRYHO LQ WKDW LW DOORZV WKH XVH RI HYHULQFUHDVLQJ XVHU
JHQHUDWHG FRQWHQW RQ WKH ZHE :H H[SHFW WKH H[SDQVLRQRI PDVKXS FRQWHQW nWK JHQHUDWLRQ FRQWHQW  DQG LWV VXSSRUWLQJ V\VWHPV WR FUHDWH DQ RSSRUWXQLW\ IRU D QHZ IRUP
RI HQWHUWDLQPHQW 5HPDLQLQJ LVVXHV VXFK DV D TXDQWLWDWLYH

5. IMPLEMENTATION OF DANCEREPRODUCER,Q WKLV VHFWLRQ ZH GHVFULEH WKH GDWDVHW XVHG DQG WULDO XVHUFRPPHQWV UHJDUGLQJ WKH V\VWHP HIIHFWLYHQHVV
5.1 Dataset
7R JHQHUDWH D GDQFH YLGHR E\ VHJPHQWLQJ DQG FRQFDWHQDWLQJ IURP H[LVWLQJ GDQFH YLGHR DQG WR PRGHO WKH YDULRXVUHODWLRQVKLSV EHWZHHQ PXVLF DQG DQ LPDJH VHTXHQFH WKH
GDWDEDVH VKRXOG IXO¿OO WKH IROORZLQJ IRXU FRQGLWLRQV&RQGLWLRQ  7KH PDLQ FRQWHQW RI YLGHR FOLSV LV GDQFH
&RQGLWLRQ  9LGHR FOLSV DUH VLPLODU W\SHV RI 0$'
PRYLHV VR WKDW WKHLU PL[WXUH JHQHUDWHG E\ RXU V\V
WHP FDQ ORRN OLNH D FRQVLVWHQW FRQWHQW KWWSZZZEDQGDLQDPFRJDPHVFRMSFVOLVWLGROPDVWHU KWWSZZZJHRFLWLHVMSKLJXFKXXLQGH[ HKWP
 KWWSZZZQLFRYLGHRMS

 'HPRQVWUDWLRQ YLGHR FOLSV JHQHUDWHG E\ RXU V\VWHP DUH DYDLODEOH DWKWWSVWDIIDLVWJRMSWQDNDQR'DQFH5H3URGXFHU3URFHHGLQJVRIWKH60&WK6RXQGDQG0XVLF&RPSXWLQJ&RQIHUHQFH-XO\3DGRYD,WDO\HYDOXDWLRQ RI WKLV V\VWHP IHDWXUH H[WUDFWLRQ IRU GDQFH PR
WLRQ LQ GHWDLO OLNH WKH ERG\ PRWLRQ GHWHFWLRQ  DQG DQ LQWHUIDFH WKDW FDQ DGMXVW PHDVXUH RU VHFWLRQ ERXQGDULHV ZLOOEH WRSLFV FRYHUHG LQ RXU IXWXUH ZRUN

>@ 0 1LVKL\DPD 7 .LWDKDUD . .RPDWDQL 7 2JDWD DQG+ * 2NXQR ³$ &RPSXWDWLRQDO 0RGHO RI &RQJUXHQF\
EHWZHHQ 0XVLF DQG 9LGHR LQ 0XOWLPHGLD &RQWHQW´ LQ
IPSJ SIG Technical Reports 2007-MUS-069 YRO 
QR   SS ± LQ -DSDQHVH

Acknowledgments>@ * 7]DQHWDNLV DQG 3 &RRN ³0XVLFDO JHQUH FODVVL¿FDWLRQ RI DXGLR VLJQDOV´ LQ IEEE Trans. on Speech andAudio Processing YRO  QR   SS ±:H WKDQN <XNL +DVHJDZD DQG 7DWVXQRUL +LUDL IRU WKHLU
KHOS
7. REFERENCES>@ 7 ; )XMLVDZD 0 7DQL 1 1DJDWD DQG + .DWD\RVH
³0XVLF PRRG YLVXDOL]DWLRQ EDVHG RQ TXDQWLWDWLYHPRGHO RI FKRUG SHUFHSWLRQ´ LQ Journal of Information
Processing Society of Japan YRO  QR   SS
± LQ -DSDQHVH>@ & /DXULHU DQG 3 +HUUHUD ³0RRG &ORXG  $ UHDOWLPH
PXVLF PRRG YLVXDOL]DWLRQ WRRO´ LQ Proc. of the 2008
Computers in Music Modeling and Retrieval Conference  SS ±>@ 0 *RWR ³$Q DXGLREDVHG UHDOWLPH EHDW WUDFNLQJ V\V
WHP IRU PXVLF ZLWK RU ZLWKRXW GUXPVRXQGV´ LQ Journal of New Music Research YRO  QR   SS±
>@ 7 6KLUDWRUL DQG . ,NHXFKL ³6\QWKHVLV RI GDQFH SHUIRUPDQFH EDVHG RQ DQDO\VHV RI KXPDQ PRWLRQ DQG PXVLF´
LQ IPSJ Transactions on Computer Vision and ImageMedia YRO  QR   SS ±>@ ;6 +XD / /X DQG +- =KDQJ ³$XWRPDWLFDOO\&RQYHUWLQJ 3KRWRJUDSKLF 6HULHV LQWR 9LGHR´ LQ Proc.
of the 12th annual ACM international conference on
Multimedia  SS ±>@ 5 &DL / =KDQJ ) -LQJ : /DL DQG :< 0D ³$XWRPDWHG 0XVLF 9LGHR *HQHUDWLRQ XVLQJ :(% ,PDJH
5HVRXUFH´ LQ Proc. of the 32nd IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP2007)  SS ,,±±,,
>@ - )RRWH 0 &RRSHUDQG DQG $ *LUJHQVRKQ ³&UHDWLQJPXVLF YLGHRV XVLQJ DXWRPDWLF PHGLD DQDO\VLV´ LQ Proc.
of the tenth ACM international conference on Multimedia  SS ±>@ ;6 +XD / /X DQG +- =KDQJ ³$XWRPDWLF PXVLF
YLGHR JHQHUDWLRQ EDVHG RQ WHPSRUDO SDWWHUQ DQDO\VLV´
LQ Proc. of the 12th annual ACM international conference on Multimedia  SS ±>@ 0 *RWR ³$ FKRUXVVHFWLRQ GHWHFWLRQ PHWKRG IRU PXVLFDO DXGLR VLJQDOV DQG LWV DSSOLFDWLRQ WR D PXVLF´ LQIEEE Trans. on Audio, Speech, and Language Processing YRO  QR   SS ±
>@ 2 *LOOHW 6 (VVLG DQG * 5LFKDUG ³2Q WKH FRUUHODWLRQRI DXGLR DQG YLVXDO VHJPHQWDWLRQV RI PXVLF YLGHRV´ LQIEEE Trans. on Circuits and Systems for Video Technology YRO  QR   SS ± (\HV:HE KWWSZZZLQIRPXVRUJ(\Z0DLQKWPO

