Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

17&,55HVHDUFK3DSHU&ODVVLILFDWLRQ([SHULPHQWVDW
+LWDFKL+LVDR0DVH

0DNRWR,ZD\DPD

+LWDFKL/WG<RVKLGDFKR7RWVXNDNX<RNRKDPDVKL.DQDJDZD-DSDQ+LWDFKL/WG+LJDVKL.RLJDNXER.RNXEXQMLVKL7RN\R-DSDQKLVDRPDVHTZ#KLWDFKLFRPPDNRWRLZD\DPDQZ#KLWDFKLFRP


$%675$&7  )RU HDFK RI WKH LGHQWLILHG ,3&V WKH UHWULHYDO VFRUHV RI WKHSDWHQWGRFXPHQWVZLWKWKH,3&DUHVXPPHG

:H UHSRUW WKH UHVXOWV RI RXU H[SHULPHQWV RQ WKH DXWRPDWLF
DVVLJQPHQW RI SDWHQW FODVVLILFDWLRQ WR UHVHDUFK SDSHU DEVWUDFWV LQ
17&,5,QPDQGDWRU\UXQVZHDSSOLHGDQDXJPHQWDWLRQRIWKH
.QHDUHVWQHLJKERUVPHWKRGVDQG³/HDUQLQJWR5DQN´WRLPSURYH
WKH FODVVLILFDWLRQ DFFXUDF\ 7KH UHVXOWV VKRZ WKDW WKHVH PHWKRGVVOLJKWO\ LPSURYH WKH FODVVLILFDWLRQ DFFXUDF\ :H DOVR FRPSDUHG
WKH DFFXUDF\ E\ WHFKQLFDO ILHOGV DQG WKH UHVXOWV VKRZ WKDW WKHDFFXUDF\GLIIHUV&DWHJRULHVDQG6XEMHFW'HVFULSWRUV+ >,QIRUPDWLRQ 6WRUDJH DQG 5HWULHYDO@ &RQWHQW $QDO\VLV
DQG,QGH[LQJ±OLQJXLVWLFSURFHVVLQJ)LUVWLQWKLVWDVNZHDSSOLHGWKH,3&VFRUHFDOFXODWLRQPHWKRGVLQ
WKH .11 PHWKRG FRQVLGHULQJ D  WKH UHWULHYDO UDQN E  FR
RFFXUUHQFH RI ,3&V DQG F  WKH VHOHFWLRQ RI . VLPLODU SDWHQWGRFXPHQWV
7KHQ ZH DSSOLHG ³/HDUQLQJ WR 5DQN´ XVLQJ WKH ,3& VFRUHV DV
IHDWXUHV :H XVHG ERWK UHVHDUFK SDSHUV DQG SDWHQW GRFXPHQWV DV
WUDLQLQJGDWD

 ,3&6&25(&$/&8/$7,21,1.11
0(7+2' 6FRUHFDOFXODWLRQXVLQJUHWULHYDOUDQN

*HQHUDO7HUPV'RFXPHQWDWLRQ.H\ZRUGVFODVVLILFDWLRQ RI UHVHDUFK SDSHUV SDWHQW UHWULHYDO NQHDUHVWQHLJKERUVPHWKRGOHDUQLQJWRUDQN

 +,7$&+,¶6$3352$&+,117&,5

2QH RI WKH VXEWDVNV LQ WKH 17&,5 3DWHQW 0LQLQJ 7DVN>@ LV WR
DVVLJQ DSSURSULDWH ,QWHUQDWLRQDO 3DWHQW &ODVVLILFDWLRQV ,3&V  WR
UHVHDUFKSDSHUDEVWUDFWVLQDIXOO\DXWRPDWLFPDQQHU:HXVHGWKH
.QHDUHVW QHLJKERUV .11  PHWKRG DV D EDVHOLQH +7&  RI
DXWRPDWLF FODVVLILFDWLRQ 2XU V\VWHP LGHQWLILHV ,3&V WKURXJK WKHIROORZLQJVWHSV 7HUPV DOOQRXQVYHUEVDQGDGMHFWLYHV DUHH[WUDFWHGIURPDQLQSXWDEVWUDFWWH[WXVLQJWKH&KDVHQ>@PRUSKRORJLFDODQDO\VLVWRRO  $ ZHLJKW IRU HDFK WHUP LV FDOFXODWHG XVLQJ D JHQHUDO WHUPIUHTXHQF\LQYHUWHGGRFXPHQWIUHTXHQF\ 7),') PHWKRG 7KHWRS.VLPLODUSDWHQWGRFXPHQWVDUHUHWULHYHGIURPDSDWHQWGRFXPHQW GDWDEDVH XVLQJ D VLPLODU GRFXPHQW UHWULHYDO HQJLQH
*(7$>@  7KH ,3&V DVVLJQHG WR HDFK RI WKH . SDWHQW GRFXPHQWV DUHLGHQWLILHG
   7KH ,3& VFRUHV DUH VRUWHG LQ GHVFHQGLQJ RUGHU 7KH WRS ;
,3&VDUHDVVLJQHGWRWKHLQSXWDEVWUDFWWH[W*(7$ LV D UHVHDUFK HIIRUW LQ WKH ³,QQRYDWLYH ,QIRUPDWLRQ7HFKQRORJ\ ,QFXEDWLRQ 3URMHFW´ SURPRWHG E\ WKH ,QIRUPDWLRQWHFKQRORJ\3URPRWLRQ$JHQF\-DSDQ ,3$ ,QRXUEDVHOLQHGHVFULEHGLQ 6HFWLRQWKH VFRUHRIDFDWHJRU\LV
FDOFXODWHGXVLQJWKHIROORZLQJIRUPXOD.

&6M ѷ :LM'㻿L ZKHUH&6MLVWKHVFRUHRIFDWHJRU\M:LMLVDIODJ LIFDWHJRU\MLV
DVVLJQHG WR GRFXPHQWL :LM  RU RWKHUZLVH :LM   '6L LV WKHUHWULHYDO VFRUH RI GRFXPHQWL DQG . LV WKH QXPEHU RI VLPLODUGRFXPHQWVXVHGLQWKH.11PHWKRG,QWKLVIRUPXODWKHUHWULHYDO
UDQN LV QRW IXOO\ UHIOHFWHG LQ ,3& VFRUHV ZKHQ WKHUH LV OLWWOH
GLIIHUHQFH EHWZHHQ WKH UHWULHYDO VFRUH RI D WRSUDQNHG GRFXPHQW
DQGWKDWRID.WKGRFXPHQWL 

7KXV ZH XVHG WKH IROORZLQJ IRUPXOD LQ WKLV PHWKRG ZKLOHFRQVLGHULQJWKHUHWULHYDOUDQNRIVLPLODUGRFXPHQWV +7& 
.
'6L
&6M ѷ
:LM ORJ UDQN

L 
L

ZKHUHUDQNLLVWKHUHWULHYDOUDQNRIGRFXPHQWL &RQVLGHULQJFDWHJRU\FRRFFXUUHQFH
,QPRVWUHVHDUFKSDSHUVWZRRUPRUH,3&VDUHDVVLJQHG7KXVZH
IRFXVHG RQ WKH FRRFFXUUHQFH RI FDWHJRULHV LQ RQH GRFXPHQW ,I
FDWHJRU\$ LV DVVLJQHG WR GRFXPHQW; DQG LI FDWHJRU\% LV DOVRDVVLJQHG WR GRFXPHQW; WKHQ FDWHJRU\$ DQG FDWHJRU\% DUHDVVLJQHGFRRFFXUUHQWO\:HFDOFXODWHGWKHUDWHRIFRRFFXUUHQFHRIWZR,3&VXVLQJ\HDUWUDLQLQJSDWHQWGRFXPHQWV:HXVHGWKHSDLUVRI,3&VZKRVHUDWH
RI FRRFFXUUHQFH LV PRUH WKDQ D JLYHQ WKUHVKROG WR WXQH WKH ,3&
VFRUH 7KH VFRUH RI ,3&$ LV DGGHG E\  LI ,3&$ LV FR

― 345 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

RFFXUUHQW WR WKH ,3&% ZKRVH UDQN LV 1WK RU KLJKHU +7& 
1RWHWKDWWKHVFRUHRI,3&VRI1WKRUKLJKHULVDOVRLQFUHDVHGE\ 6HOHFWLRQRI.VLPLODUSDWHQWGRFXPHQWV
,Q WKH FRQYHQWLRQDO .11 PHWKRG WRS . VLPLODU GRFXPHQWV DUH
XVHG WR LGHQWLI\ FDWHJRULHV ,Q WKLV FDVH EHFDXVH WKH QXPEHU RI
FDWHJRULHVRXWSXWLVOLPLWHGVRPHFDWHJRULHVWREHDVVLJQHGFDQQRWEHRXWSXW,IWKHYDOXHRI.LQFUHDVHVWRLPSURYHUHFDOOSUHFLVLRQGHFUHDVHV
7KXV LQ WKLV PHWKRG ZH G\QDPLFDOO\ FKDQJHG . GRFXPHQWV WR
LPSURYHUHFDOOZKLOHPDLQWDLQLQJSUHFLVLRQ +7& :HXVHGWKH
IROORZLQJDOJRULWKP VLPLODUSDWHQWGRFXPHQWVWRDTXHU\SDSHUDEVWUDFWDUH
UHWULHYHG

E 7KHWLWOHDQGERG\RIDEVWUDFWV  4XHU\GDWDIRUWUDLQLQJD 7KHWLWOHDQGERG\RIDEVWUDFWVE 7KHDEVWUDFWRISDWHQWGRFXPHQWV:HXVHGNLQGVRIIHDWXUHVHWV +7&WKURXJK+7& VKRZQLQ 7DEOH  )HDWXUH  LV WKH ,3& VFRUH REWDLQHG E\ WKH .11PHWKRGLQZKLFKWKHWKUHHPHWKRGV GHVFULEHGLQ6HFWLRQ DQGZHUHDSSOLHGWRWKHEDVHOLQHPHWKRGGHVFULEHGLQ6HFWLRQ 7KH PHWKRG GHVFULEHG LQ 6HFWLRQ  ZDV DOVR DSSOLHG WR
IHDWXUHVWKURXJK
,Q +7& ZH ILUVW DVVLJQHG WKH PDLQ FODVV WR D TXHU\ SDSHUDEVWUDFW 7KHQ WKH PRGHO GDWD FRUUHVSRQGLQJ WR WKH WRSUDQNHG
PDLQFODVVZDVXVHGLQOHDUQLQJWRUDQNWRGHFLGHWKHILQDOUDQN 7RS.GRFXPHQWVDUHVHOHFWHGIURPGRFXPHQWV

7DEOH)HDWXUHVHWXVHGLQOHDUQLQJWRUDQN 1FDWHJRULHVDUHLGHQWLILHGXVLQJWKH.11PHWKRG  1 FDWHJRULHV DUH GHOHWHG IURP WKH FDWHJRU\ OLVW RI VLPLODUSDWHQW GRFXPHQWV ,I QR FDWHJRU\ UHPDLQV LQ D GRFXPHQWWKHGRFXPHQWLVGHOHWHG  6WHS   WKURXJK VWHS   DUH LWHUDWHG XQWLO WKH QXPEHU RI
FDWHJRULHVLGHQWLILHGLQVWHS  UHDFKHV

 /($51,1*725$1.%$6('21.11
0(7+2' 7UDLQLQJGDWD
:H DSSOLHG WKH ³/HDUQLQJ WR 5DQN´ DSSURDFK WR UHVHDUFK SDSHUFODVVLILFDWLRQ 6XIILFLHQW WUDLQLQJ GDWD LV QHFHVVDU\ IRU WKLVDSSURDFK +RZHYHU ZH FRXOG RQO\ XVH  UHVHDUFK SDSHUDEVWUDFWV 17&,5 GU\ UXQ GDWD 17&,5 IRUPDO UXQ GDWD DQG17&,5GU\UXQGDWD :HDSSOLHGUHVHDUFKSDSHUDEVWUDFWVXVHGLQWKH17&,5GU\UXQ DV WUDLQLQJ GDWD RI OHDUQLQJ WR UDQN DQG  DEVWUDFWV ZHUH
XVHGIRUWKHHYDOXDWLRQ
:H DOVR XVHG SDWHQW GRFXPHQWV IRU TXHULHV LQ WKH WUDLQLQJ :H
VHOHFWHGSDWHQWGRFXPHQWVDVWUDLQLQJTXHULHVIRUHDFKPDLQFODVV:HJHQHUDWHGPRGHOGDWDIRUHDFKPDLQFODVV

4XHU\GDWD
IRUWUDLQLQJ

+7&
+7&
+7&
+7&

3DSHUVGRFV 


ż
ż
ż
ż


ż

ż
ż



ż
ż
ż

)HDWXUHV 








ż
ż













3DSHUV
ż
ż
ż
ż
ż


GRFV 
3DWHQW
+7&
DEVWUDFWV





ż
ż
GRFV +7&

6FRUHPHUJLQJRI+7&DQG+7&,3&VFRUHREWDLQHGE\.11PHWKRGZLWKWKHIROORZLQJSDUDPHWHUVTWH WLWOH DEVWUDFWUW IXOOWH[WRISDWHQWV  ,') IXOOWH[WRISDWHQWVTWH WLWOH    UW IXOOWH[WRISDWHQWV  ,') IXOOWH[WRISDWHQWVTWH WLWOH    UW DEVWUDFWRISDWHQWV ,') IXOOWH[WRISDWHQWVTWH WLWOH DEVWUDFWUW DEVWUDFWRISDWHQWV ,') IXOOWH[WRISDWHQWVTWH DEVWUDFW   UW DEVWUDFWRISDWHQWV ,') IXOOWH[WRISDWHQWVTWH DEVWUDFW   UW IXOOWH[WRISDWHQWV  ,') SDSHUDEVWUDFWTWH DEVWUDFW   UW DEVWUDFWRISDWHQWV ,') SDSHUDEVWUDFW
127(TWH VFRSHRITXHU\WHUPH[WUDFWLRQUW UHWULHYDOWDUJHW
+7&

 7RROV:HXVHGIUHHVRIWZDUH690UDQN>@RUWKH6XSSRUW9HFWRU0DFKLQH
IRU5DQNLQJGHYHORSHGE\&RUQHOO8QLYHUVLW\ (9$/8$7,21

 )HDWXUHV
7KH IHDWXUHV XVHG LQ OHDUQLQJ WR UDQN ZHUH OLPLWHG LQ WKHPDQGDWRU\ UXQ 7KH ELEOLRJUDSKLF GDWD RQ IRU H[DPSOH DXWKRUVSXEOLFDWLRQ GDWHV DQG VRXUFHV FRXOG QRW EH XVHG 7KXV ZH XVHG
,3&VFRUHVREWDLQHGE\WKH.11PHWKRGZLWKWKHFRPELQDWLRQRIWKHIROORZLQJIRXUNLQGVRISDUDPHWHUV  7KHVFRSHRITXHU\WHUPH[WUDFWLRQD 2QO\WKHWLWOH
E 2QO\WKHERG\RIWKHDEVWUDFWF %RWKWKHWLWOHDQGERG\RIWKHDEVWUDFW:H HYDOXDWHG WKH HIIHFW RI WKH PHWKRGV XVLQJ  17&,5
IRUPDOUXQTXHULHV17&,5GU\UXQTXHULHVDQG17&,5 IRUPDO UXQTXHULHV7KH UHVXOWV DUH VKRZQ LQ7DEOHDQG ,3& VFRUH FDOFXODWLRQ PHWKRGV +7& +7& DQG +7& 
GHVFULEHG LQ 6HFWLRQ  ZHUH VOLJKWO\ HIIHFWLYH LQ D FRPSDULVRQZLWKWKHEDVHOLQH +7& LQDOOTXHU\GDWDVHWV  5HWULHYDOWDUJHWD )XOOWH[WRISDWHQWGRFXPHQWV
E $EVWUDFWRISDWHQWGRFXPHQWV+RZHYHU WKH HIIHFW RI OHDUQLQJ WR UDQN YDULHV ,W LV VOLJKWO\HIIHFWLYHLQWKHVXEJURXSDQGPDLQJURXSEXWQRWHIIHFWLYHLQWKH
VXEFODVV ,Q WKH VXEJURXS LW LV RQO\ VRPHZKDW HIIHFWLYH LQ
17&,5 IRUPDO UXQ TXHULHV DQG 17&,5 IRUPDO UXQ TXHULHVZKLOHLWLVQRWHIIHFWLYHLQ17&,5GU\UXQTXHULHV)XUWKHUPRUH
XVLQJ SDWHQW GRFXPHQWV DV TXHULHV IRU WUDLQLQJ LV PRUH HIIHFWLYH
WKDQXVLQJSDSHUDEVWUDFWV  ,')FDOFXODWLRQD )XOOWH[WRISDWHQWGRFXPHQWV

7DEOHVKRZVWKH0$3VDFFRUGLQJWR,3&6HFWLRQ TXHU\GDWDVHWV
RI17&,5IRUPDOUXQ17&,5GU\UXQDQG17&,5IRUPDO

― 346 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

7DEOH(YDOXDWLRQUHVXOWVE\6HFWLRQ VXEJURXS 

7DEOH(YDOXDWLRQUHVXOWV VXEJURXS ,'+7&
+7&
+7&
+7&
+7&
+7&
+7&
+7&
+7&
+7&
+7&)RUPDO
TXHULHV









0$3
'U\
TXHULHV








,')RUPDO
TXHULHV











+7&
+7&
+7&
+7&
+7&
+7&
+7&
+7&
+7&
+7&
+7&




7DEOH(YDOXDWLRQUHVXOWV PDLQJURXS ,'+7&
+7&
+7&
+7&
+7&
+7&
+7&
+7&
+7&
+7&
+7&)RUPDO
TXHULHV










0$3
'U\
TXHULHV







)RUPDO
TXHULHV











7DEOH(YDOXDWLRQUHVXOWV VXEFODVV ,'+7&
+7&
+7&
+7&
+7&
+7&
+7&
+7&
+7&
+7&
+7&)RUPDOTXHULHV









0$3
'U\
TXHULHV






)RUPDOTXHULHV











0$3
$%'()㹁㹅㹆,3&V ,3&V ,3&V ,3&V






































UXQ ZHUH XVHG IRU WKLV DQDO\VLV  ,3& VFRUH FDOFXODWLRQ PHWKRGVDUHHIIHFWLYHLQDOO6HFWLRQV/HDUQLQJWRUDQNLQ6HFWLRQ*LVQRW
HIIHFWLYH EXW LW LV VOLJKWO\ HIIHFWLYH LQ RWKHU VHFWLRQV ,Q WKH17&,5IRUPDOUXQTXHU\GDWDVHWRI,3&VDVVLJQHGWRWKH
TXHULHV EHORQJ WR 6HFWLRQ* DQG  EHORQJ WR 6HFWLRQ*ZKLFKDIIHFWVWKHRYHUDOODFFXUDF\ &21&/86,216
:H DSSOLHG ,3& VFRUH FDOFXODWLRQ PHWKRGV LQ WKH .11 PHWKRGFRQVLGHULQJ WKH UHWULHYDO UDQN FRRFFXUUHQFH RI ,3&V DQG WKHVHOHFWLRQ RI . VLPLODU SDWHQW GRFXPHQWV 7KHVH PHWKRGVFRQWULEXWHG WR VOLJKWO\ LPSURYLQJ WKH DFFXUDF\ :H DOVR DSSOLHG
WKHOHDUQLQJWRUDQNDSSURDFKWRWKH.11PHWKRG7KLVDSSURDFKLVVOLJKWO\HIIHFWLYHLQVRPHWHFKQLFDOILHOGVEXWQRWLQRWKHUV:HVKRXOGWDNHWKHFKDUDFWHULVWLFVRIWHFKQLFDOILHOGVLQWRDFFRXQW
8VLQJ WKH ZKROH WH[W RI D UHVHDUFK SDSHU WR DVVLJQ ,3&V ZRXOG
PDNHIRULQWHUHVWLQJIXWXUHZRUN8VLQJELEOLRJUDSKLFGDWDVXFKDVDXWKRU QDPHV DXWKRU DIILOLDWLRQV DQG SXEOLFDWLRQ GDWHV DQGVRXUFHV WR LPSURYH FODVVLILFDWLRQ DFFXUDF\ LQ OHDUQLQJ WR UDQNFRXOGDOVR\LHOGVRPHLQWHUHVWLQJUHVXOWV 5()(5(1&(6>@ +LGHWVXJX1DQED$WVXVKL)XMLL0DNRWR,ZD\DPD7DLLFKL+DVKLPRWR2YHUYLHZRIWKH3DWHQW0LQLQJ7DVNDWWKH
17&,5:RUNVKRS3URFHHGLQJVRIWKHWK17&,5:RUNVKRS0HHWLQJRQ(YDOXDWLRQRI,QIRUPDWLRQ$FFHVV
7HFKQRORJLHV,QIRUPDWLRQ5HWULHYDO4XHVWLRQ$QVZHULQJDQG&URVVOLQJXDO,QIRUPDWLRQ$FFHVV
>@ &KDVHQKWWSFKDVHQQDLVWMSKLNL&KD6HQ>@ *(7$KWWSJHWDH[QLLDFMSHLQGH[KWPO>@ 690UDQNKWWSZZZFVFRUQHOOHGX3HRSOHWMVYPBOLJKWVYPBUDQNKWPO






― 347 ―