言語処理学会 第24回年次大会 発表論文集 (2018年3月)

ґଘߏ଄ͷ࿈࠯Λߟྀͨ͠χϡʔϥϧจѹॖ্֞֎ ӳ߶ ྛ ࠀ඙ ฏඌ ౒ Ӭా ণ໌NTT ίϛϡχέʔγϣϯՊֶ‫ॴڀݚૅج‬1 ͸͡Ίʹจѹॖͱ͸ɺ‫ݪ‬จͷେҙΛอ࣋ͭͭ͠ɺ୯‫ޠ‬Λ࡟আ͢Δ͜ͱͰ୹͍จΛੜ੒͢ΔλεΫͰ͋Δɻ‫ࡏݱ‬ɺѹॖจͷՄಡੑΛ୲อ͢ΔͨΊʹɺߏจ໦Λ‫מ‬ΓࠐΉख๏ [6, 9, 1, 3] ΍ɺSequence-to-Sequence (Seq2Seq) Λ༻͍Δख๏ [2] ͕ఏҊ͞Ε͍ͯΔɻWang Β [11] ͸͜ΕΒͷख๏Λ૊Έ߹ΘͤΔ͜ͱͰ࠷ߴੑೳΛୡ੒͢Δจѹॖ‫ث‬Λ࣮‫ͨ͠ݱ‬ɻ۩ମతʹ͸ɺ
LSTM Λ༻͍ͯจதͷ֤୯‫͕ޠ‬ѹॖจʹ࠾༻͞ΕΔॏΈ (֬཰஋) Λ෇༩͠ɺॏΈͷ࿨͕࠷େͱͳΔΑ͏୯‫ޠ‬ґଘߏ଄໦Λ‫מ‬ΓࠐΉɻ͔͠͠ɺWang Βͷख๏͸ɺґଘߏ଄໦ͷ༿ϊʔυ͔Βॱʹ୯‫ޠ‬Λ࡟আ͢Δख๏Ͱ͋Γɺґଘߏ଄໦ͷ༿ʹ͍ۙϊʔυʹॏཁͳ୯‫ͯ͠ͱ͕ͨͬ͋ޠ‬΋ɺͦΕ͕͔ࠜΒ཭Ε͍ͯΔɺͭ·Γਂ͍Ґஔʹ͋Δ৔߹ʹ͸ɺѹॖจʹ࢒͢͜ͱ͕೉͍͠ɻ
·ͨɺਤ 1 ʹࣔ͢Α͏ʹɺࣄલௐࠪͷ݁ՌɺSeq2Seq ͷΑ͏ͳґଘߏ଄໦ʹཔΒͳ͍ख๏Ͱ͋ͬͯ΋ɺਂ͍ґଘߏ଄Λ࣋ͭจͰ͸จѹॖͷੑೳ͕௿Լ͢Δ͜ͱ͕൑໌ͨ͠*1 ɻ͞Βʹਤ 2 ʹࣔ͢Α͏ʹɺ͜ͷΑ͏ͳਂ͍ґଘߏ଄໦͸௕จʹଘࡏ͢Δ͜ͱ͕ଟ͍ɻ௕จΛ୹͘͢Δ͜ͱ͕໨తͰ͋ΔจѹॖλεΫʹ͓͍ͯɺਂ͍ґଘߏ଄໦Λ࣋ͭจͷѹॖੑೳΛ޲্ͤ͞Δ͜ͱ͸ɺॏཁͳ՝୊Ͱ͋Δɻ
ਤ 3 ʹਂ͍ґଘߏ଄Λ࣋ͭจͱͦͷѹॖ݁ՌͷྫΛࣔ͢ɻ͜ͷྫ͸ిྗ༌ೖ‫ڠ‬ఆʹؔ͢ΔจͰ͋Γɺࠃ໊͕ॏཁͰ͋
Δɻࠃ໊Λࣔ۟͢ “from Kyrgyz Republic and Tajikistan” Λҡ࣋ͭͭ͠ѹॖΛߦ͏ʹ͸ɺ͜ͷ͔۟Βϧʔτ·Ͱʹ୧Δ୯‫ޠ‬“signed”ɺ“resolution”ɺ“import” ΋ѹॖ‫ޙ‬ͷจͰҡ࣋͠ͳ͚Ε͹ͳΒͳ͍ɻͭ·Γɺґଘߏ଄໦ͷਂ͍Ґஔʹॏཁͳ୯‫ޠ‬͕ग़‫͢ݱ‬Δ৔߹ʹɺͦΕΛѹॖจʹ࢒ͨ͢Ίʹ͸ґଘߏ଄ͷ࿈࠯Λߟྀ͠ͳ͚Ε͹ͳΒͳ͍ɻຊߘͰ͸ɺਂ͍ґଘߏ଄Λ࣋ͭจͷѹॖਫ਼౓Λ޲্ͤ͞ΔͨΊʹɺ೚ҙͷ୯‫͔ޠ‬Β૆ઌ΁ͷґଘߏ଄ͷ࿈࠯Λߟྀͨ͠஫ҙ‫ߏػ‬ΛఏҊ͠ɺSeq2Seq ʹ૊ΈࠐΉɻ͜ͷΑ͏ͳ࿈࠯Λ஫ҙ‫ྀ͢ߟͰߏػ‬ΔͨΊʹ͸ɺ‫ࢣڭ‬σʔλͱͯ͠ਖ਼ղͱͳΔґଘߏ଄໦͕ඞཁͱͳΔɻ͔͠͠ɺզʑ͕࢖༻Մೳͳ‫ࢣڭ‬σʔλ͸‫ط‬ଘͷߏจղੳ‫ʹث‬Αͬͯղੳ͞Εͨ݁Ռʹ‫ݶ‬ΒΕΔͨΊɺ‫ޡ‬ΓΛ‫ؚ‬ΉՄೳੑ͕͋Δɻ‫ޡ‬ΓͷӨ‫ڹ‬Λආ͚ΔͨΊɺఏҊख๏͸ɺෳ਺ͷ୯‫ؒޠ‬ͷґଘͷՄೳੑΛߟྀͭͭ͠ɺ୯‫ޠ‬ͷ࡟আ֬཰Λಉ࣌ʹֶश͢Δɻ࣮‫ݧ‬ͷ݁ՌɺఏҊख๏͸ Google จѹॖσʔληοτʹ͓͍ͯ F ஋Ͱ࠷ߴਫ਼౓Λୡ੒͠ɺROUGE-1,2,L είΞʹ͓͍ͯ΋ैདྷख๏͔Βͷվળ͕֬ೝ͞Εͨɻ͞ΒʹɺධՁͷର৅ਤ 1: ґଘߏ଄໦ͷਂ͞ͱ Seq2Seq ͷ F ஋

ਤ 2: จͷ௕͞ͱґଘߏ଄໦ͷਂ͞Λਂ͍ґଘߏ଄Λ࣋ͭจʹ‫ݶ‬ఆͨ͠৔߹ʹ͓͍ͯ΋ɺఏҊख๏ʹΑΔ F ஋ͱ ROUGE-1,2,L είΞͷվળ͕֬ೝ͞Εͨɻ

2Seq2Seq ʹΑΔจѹॖզʑ͕ఏҊ͢Δ஫ҙ‫ߏػ‬Λ૊ΈࠐΉ Seq2Seq Ϟσϧʹ͍ͭͯઆ໌͢Δɻ͜ͷ Seq2Seq Ϟσϧ͸ɺFillipova ͕ఏҊͨ͠Ϟσϧͷ֦ுͰ͋Γɺ୯‫ޠ‬ຒΊࠐΈ૚ɺΤϯίʔμɺσίʔμɺग़ྗ෦ʹΑͬͯߏ੒͞Ε͍ͯΔɻจѹॖ͸ೖྗ୯‫ ྻޠ‬x = (x0 , ..., xn ) தͷ୯‫ ޠ‬xt ʹର͠ɺग़ྗϥϕϧ yt Λ (“ҡ࣋”ɺ“࡟আ”ɺ“จͷऴྃ”) ͔Βબ୒͢ΔλεΫͰ͋Δͱߟ͑Δ͜ͱ͕Ͱ͖ΔɻFillipova Β [2] ͸ϥϕϧͷ༧ଌʹ Seq2Seq Λ༻͍Δ͜ͱͰɺग़ྗϥϕϧؒͷґଘΛߟྀͨ͠จѹॖΛߦ͍ͬͯΔɻ୯‫ޠ‬ຒΊࠐΈ૚Ͱ͸ɺ୯‫ ޠ‬xt ʹର͢ΔຒΊࠐΈϕΫτϧ etΛ‫͢ࢉܭ‬ΔͨΊʹɺ୯‫ޠ‬ͷද૚ɺ඼ࢺɺґଘؔ܎ϥϕϧʹର
͢ΔຒΊࠐΈϕΫτϧ wi ɺpi ɺri Λ ei = [wi , pi , ri ] ͷΑ͏ʹ݁߹͢Δ [11]ɻ ͳ͓ɺ[] ͸ϕΫτϧͷ݁߹Λද͢ɻΤϯίʔμͰ͸ɺforward-LSTM ͱ backward-LSTM ͔Βߏ੒͞ΕΔ྆ํ޲ LSTM Λ༻͍ͯɺ‫͞ࢉܭ‬ΕͨຒΊࠐΈϕΫτϧ et ΛӅΕঢ়ଶϕΫτϧ ht ΁ͱม‫͢׵‬Δɻbackward-LSTMͷ࠷ऴঢ়ଶ͸σίʔμͷॳ‫ظ‬ঢ়ଶ΁ͱҾ͖‫͕ܧ‬ΕΔɻσίʔμͰ͸ɺ୯‫ޠ‬ຒΊࠐΈ et ɺyt−1 ͷ 3-bit ͔ΒͳΔone-hot ͳϕΫτϧຒΊࠐΈͱҎલͷӅΕঢ়ଶ dt−1 (‫ޙ‬ड़) Λ→st݁߹ͨ͠ϕΫτϧΛɺforward-LSTM Λ༻͍ͯɺӅΕঢ়ଶ −΁ͱม‫͢׵‬Δɻग़ྗ෦Ͱ͸ɺϥϕϧ yt ͷग़ྗ֬཰Λɺ

*1 F ஋ͷ‫ʹࢉܭ‬͸ Fillipova Βͷ Seq2Seq [2] Λɺୈ 4 અͰઆ໌͢Δ࣮‫ݧ‬ઃఆʹ‫ͨ͠༻࢖͍ͯͮج‬ɻ― 1096 ―P (yt | y<t , x)=sof tmax(Wo · dt ) · δyt ,→s t]
dt =[ht , −(1)(2)

Copyright(C) 2018 The Association for Natural Language Processing.
All Rights Reserved.

Pakistan signed a resolution on Monday to import 1,300 MW of electricity from Kyrgyz Republic and Tajikistan to overcome power shortage in summer season ...

ਤ 3: ਂ͍ґଘߏ଄໦Λ࣋ͭจͱͦͷѹॖ݁Ռͷྫɻփ৭ͷ୯‫ޠ‬͸࡟আ͞Εͨ͜ͱΛ͍ࣔͯ͠Δɻ

y1

y6y7

sof tmax

sof tmax

sof tmax

h1
←
−h0

Ω1

h7

Ω6

h6−
→
s1−→
s6−→
s7

y0

y5

y6x1x6࠶‫ؼ‬త஫ҙ‫( ߏػ‬d = {1, 2, 3})d=3γ3,7Ω7x7γ2,7Σબ୒
త஫
ҙ‫ػ‬
ߏ

β2,7 · γ2,7

Σα1,5,7 · h5

d=1γ1,73. બ୒త஫ҙ‫ߏػ‬͸֤ γd,t ͷ࣍਺ d ∈ d ʹର͢ΔॏΈ βd,tΛ‫͢ࢉܭ‬Δɻd ͸࣍਺ͷू߹Λද͢ɻβd,t ͸ΤϯίʔμͱσίʔμͷӅΕঢ়ଶ͔Β‫͞ࢉܭ‬ΕΔɻͦΕͧΕͷ βd,t ·γd,tͷ૯࿨ Ωt ͕બ୒త஫ҙ‫ߏػ‬ͷग़ྗͱͳΔɻ

β3,7 · γ3,7Σ

α1,2,7 · h2

Pparent d = 2·Ͱ୧ͬͨ৔߹ɺ୯‫ ޠ‬x6 ɺx5 ɺx2 ͕ͦΕͧΕ 1 ࣍ɺ2 ࣍ɺ3 ࣍ͷ୯‫ ޠ‬x7 ͷ਌ͱͳΔɻ͜ΕΒͷ਌ʹରԠ͢ΔӅΕঢ়ଶ h6 ɺh5 ɺh2 ͸ͦΕͧΕ 1 ࣍ɺ2 ࣍ɺ3 ࣍ͷ஫ҙ α1,7,6 ɺ
α2,7,5 ɺα3,7,2 ʹΑͬͯॏΈ෇͚͞Εͨ‫ޙ‬ɺ଍͠߹Θ͞Εɺબ୒త஫ҙ‫ߏػ‬ͷೖྗ γd,t ͱͳΔɻβ1,7 · γ1,7

Σ4. ग़ྗ૚ʹ͓͍ͯ Ωt ͱΤϯίʔμɺσίʔμͷӅΕঢ়ଶ͕݁߹͞Εɺ֤ϥϕϧͷग़ྗ֬཰͕‫͞ࢉܭ‬ΕΔɻ֤஫ҙ‫ߏػ‬ͷৄࡉͱ໨తؔ਺ͷৄࡉʹ͍ͭͯ͸ҎԼͷઅͰઆ໌͞ΕΔɻ

α1,6,7 · h6

3.1h1h2h3h4

h5h6h7−→ ←− −→h7 h0 s7ఏҊख๏͸ɺZhang Β [12] ΍ Hashimoto Β [5] ͱಉ͘͡ɺґଘߏ଄Λ஫ҙ‫্ߏػ‬ͷάϥϑͱͯ͠ද‫͍ͯ͠ݱ‬Δɻґଘߏ଄໦Ͱ͸ࢠͱͳΔ୯‫ޠ‬͸ 1 ͭͷ਌ͱͳΔ୯‫ޠ‬Λ࣋ͭɻ͜ͷ੍໿ͷԼͰɺґଘߏ଄ղੳ͸࣍ͷΑ͏ʹද͞ΕΔɻจ x =(x0 , x1 , ..., xn ) ͕༩͑ΒΕͨ࣌ɺt ̸= 0 Λຬͨ͢୯‫ ޠ‬xt ͷ਌͸ɺi ̸= t Λຬͨ͢୯‫ ޠ‬xi ͔Βબ୒͞ΕΔɻͳ͓ x0 ͸ϧʔτϊʔυΛද͢ɻxt ͷ਌͕ xi ͱͳΔ֬཰͸ॏΈߦྻ Wg ɺva ɺUa ɺWa Λ༻͍ͯ਌΁ͷ஫ҙ‫ߏػ‬ROOTh0h1h2

h3h4h5h6h7x0x1x2x3x4x5x6x7

਌΁ͷ஫ҙ‫ߏػ‬ਤ 4: ఏҊख๏ͷωοτϫʔΫߏ੒ਤͷΑ͏ʹ‫͢ࢉܭ‬Δ [10]ɻWo ͸ιϑτϚοΫε૚ͷॏΈߦྻΛɺδyt ͸ yt ൪໨ͷཁૉͰ͸ 1 ΛɺͦΕҎ֎Ͱ͸ 0 Λग़ྗ͢Δ 2 ஋ϕΫτϧΛͦΕͧΕද͢ɻ

3 ఏҊख๏
զʑ͕ఏҊ͢Δґଘߏ଄ͷ࿈࠯Λߟྀͨ͠஫ҙ‫ߏػ‬͸ɺલઅʹͯઆ໌ͨ͠ Seq2Seq ͷ্ʹߏங͞ΕΔɻਤ 4 ͸ఏҊख๏Λ༻͍ͨ Seq2Seq ͷ֤ϞδϡʔϧͱωοτϫʔΫؒͷ݁߹Λ͍ࣔͯ͠Δɻ͜ͷਤͰ͸ɺೖྗ͞Εͨ୯‫ʹྻޠ‬ର͠ɺ୯‫ ޠ‬x7ͷϥϕϧ y7 ͷग़ྗ֬཰Λܾఆ͢Δաఔ͕ඳ͔Ε͍ͯΔɻy7ͷग़ྗ֬཰͸࣍ͷΑ͏ʹ༧ଌ͞ΕΔɻ

1. ਌ ΁ ͷ ஫ ҙ ‫ ߏ ػ‬͸ ୯ ‫ ޠ‬xt ͷ ਌ ͕ xi ͱ ͳ Δ ֬ ཰
Pparent (xi |xt , x) ΛӅΕঢ়ଶ hi ͱ ht Λ༻͍ͯ‫͢ࢉܭ‬Δɻ͜ͷ‫ࢉܭ‬͸શͯͷ xi , xt ͷରʹରͯ͠ߦΘΕΔɻਤ 4 ʹ͓͚Δ਌΁ͷ஫ҙ‫ߏػ‬தͷ‫ހ‬͸ɺͦΕͧΕͷࢠʹର͠ɺ਌ͱͳΔ֬཰͕΋ͬͱ΋ߴ͍୯‫ޠ‬Λ͍ࣔͯ͠Δɻ

2. ࠶‫ؼ‬త஫ҙ‫ߏػ‬͸ xt ͷ d ࣍ͷ਌͕ xj ͱͳΔ֬཰ αd,t,jΛɺ࠶‫ؼ‬తʹ Pparent (xi |xt , x) Λ༻͍Δ͜ͱͰ‫͢ࢉܭ‬Δɻ
αd,t,j ͸ d ࣍ͷ਌Λߟྀͨ͠ࡍͷӅΕঢ়ଶ h ͷॏΈ෇͖࿨ γd,t Λ‫͢ࢉܭ‬ΔͨΊͷ஫ҙͱͯ͠ѻΘΕΔɻਤ 4 ʹ͓͍ͯɺ୯‫ ޠ‬x7 ͷ਌ͱͳΔ֬཰͕΋ͬͱ΋ߴ͍୯‫ޠ‬Λ 3 ࣍

Pparent (xi |xt , x)=sof tmax(Wg · g(hj , ht )) · δxj ,g(hj , ht )=vaT · tanh(Ua · hj + Wa · ht ),(3)(4)ͷΑ͏ʹ‫͞ࢉܭ‬ΕΔɻߏจղੳ‫ͱث‬͸ҟͳΓɺఏҊख๏Ͱ͸ґଘߏ଄໦Λ֬ఆ͢Δ͜ͱ͸ͤͣɺPparent (xi |xt , x) Λͦͷ
··࢖༻͍ͯ͠Δɻ

3.2

࠶‫ؼ‬త஫ҙ‫ߏػ‬࠶‫ؼ‬త஫ҙ‫ߏػ‬͸ɺxt ͷ d ࣍ͷ਌͕ xj ͱͳΔ֬཰Λ

αd,t,j =

⎧ n⎪
⎨ %⎪
⎩ k=1
P

αd−1,t,k · α1,k,jparent (xi |xt , x)(d > 1)(5)(d = 1)ͷΑ͏ʹ࠶‫ؼ‬తʹ‫͢ࢉܭ‬Δɻ·ͨɺϧʔτϊʔυ͸਌Λ࣋ͨͣɺࢠͱ਌͕ಉ͡୯‫ͳʹޠ‬Δ͜ͱ͸ͳ͍ͱ͍͏ґଘߏ଄໦ͷ੍໿Λ஫ҙ‫ʹ্ߏػ‬औΓࠐΉͨΊʹɺզʑ͸ α1,t,j ʹର͠ɺ

α1,t,j =&100(t = 0 ∧ j = 0)(t = 0 ∧ j > 0)(t ̸= 0 ∧ t = j)(6)ͷΑ͏ͳ੍໿Λ՝ͨ͠ɻࣜ (6) ͷ 1 ߦ໨ͱ 2 ߦ໨ΑΓϧʔτϊʔυͷ਌͸ϧʔτϊʔυͱͳΔɻ͜ΕΒͷ੍໿͸ϧʔτϊʔυ͕਌Λ࣋ͨͳ͍ͱ͍͏͜ͱΛද͍ͯ͠Δɻࣜ (6) ͷ 3― 1097 ―Copyright(C) 2018 The Association for Natural Language Processing.
All Rights Reserved.

ALL

DEPTH

ROUGE

F1

∆C

1

2

L

ROUGE

F1

∆C

1

2

L

TaggerTagger+ILP

82.8
79.0

81.1
76.1

72.4
64.6

80.9
75.8

-3.0
-4.1

82.8
77.5

80.6
74.7

72.2
64.1

80.3
74.3

-3.2
-4.6

Bi-LSTM
Bi-LSTM-Dep
Attn
Base

81.9
82.3
82.4
82.7

81.1
81.5
81.6
81.9

73.7
74.174.3
74.7

80.9
81.3
81.4
81.7

-2.2
-2.1
-2.3
-2.4

81.3
81.5
81.9
82.1

80.4
80.7
81.0
81.0

73.3
73.5
73.9
73.9

80.1
80.3
80.6
80.7

-2.2
-2.1
-2.3
-2.5

83.0
83.1
83.2

81.7
82.5
82.9

74.5
75.2
75.8

81.5
82.2
82.7

-2.9
-2.1
−1.7

82.7
82.6
82.8

81.4
81.8
82.2

74.5
74.9
75.5

81.1
81.5
82.0

-2.8
-2.1
−2.0

Prop

(d = {1})(d = {1, 2})(d = {1, 2, 4})ද 1: ࣗಈධՁͷ݁Ռɻ ALL ͱ DEPTH ͸ɺ ͦΕͧΕશͯͷจͰͷධՁͱɺਂ͍໦ (ਂ͕͞ςετσʔλશମͰͷฏ‫ۉ‬஋Ͱ͋Δ 8 Ҏ্ͷ໦) Λ࣋ͭจͰͷධՁ݁ՌΛද͍ͯ͠Δɻd ͸࢖༻ͨ࣍͠਺ͷू߹Λද͍ͯ͠Δɻߦ໨͸਌ͱࢠ͕ಉ͡୯‫ͳʹޠ‬Δ͜ͱΛ๷͍Ͱ͍Δɻࣜ (5) ͷ1 ߦ໨͕ߦྻੵͷఆٛΛຬͨͨ͢Ίʹɺ͜ͷ࠶‫ؼ‬తͳ஫ҙͷ‫ࢉܭ‬͸ CPU ‫ ͼٴ‬GPU ্ʹ͓͍ͯޮ཰తʹ‫͢ࢉܭ‬Δ͜ͱ͕ՄೳͰ͋Δɻ͜ͷΑ͏ʹ࠶‫ؼ‬తʹ஫ҙΛ༻͍Δ͜ͱʹΑͬͯɺߴ࣍ͷ਌ͷ֬཰Λ‫͢ࢉܭ‬ΔࡍʹɺͦΕͧΕͷ਌͝ͱʹ஫ҙ‫ߏػ‬Λ४උ͢Δඞཁ͕ͳ͘ͳΔɻ·ͨɺͦΕʹΑΓɺෳ਺ͷ஫ҙΛֶश͢Δඞཁ͕ͳ͘ͳΔͨΊɺֶश࣌ʹɺ֤஫ҙͷ෼෍ͷॏΈΛௐ੔͢ΔͨΊͷϋΠύʔύϥϝʔλ΋ಉ༷ʹෆཁͱͳΔɻͦͷ্ɺ͜ͷख๏ʹΑΓߴ࣍ͷґଘؔ܎Λ௚઀ߟྀ͢Δඞཁ͕
ͳ͘ͳΔͨΊɺֶशσʔλ͕εύʔεʹͳΔ͜ͱΛ๷͙͜ͱ
͕Ͱ͖Δɻ্‫͞ࢉܭͰه‬Εͨ αd,t,j ͸ҎԼͷΑ͏ʹɺ

γtd =n%k=1(7)αd,t,k · hk ,Τϯίʔμͷ྆ํ޲ LSTM ͷӅΕঢ়ଶ h ʹର͢ΔॏΈ෇͖࿨ͷ‫͍༻ʹࢉܭ‬ΒΕΔɻ‫͞ࢉܭ‬Εͨ γtd ͸બ୒త஫ҙ‫ߏػ‬ͷೖྗʹҾ͖‫͕ܧ‬ΕΔɻ

3.3 બ୒త஫ҙ‫ߏػ‬ೖྗจʹର͠ɺద੾ͳ࣍਺ͷґଘؔ܎Λߟྀ͢ΔͨΊʹɺબ୒త஫ҙ‫ߏػ‬͸֤ӅΕঢ়ଶ γtd ͷॏΈ෇͚࿨ Ωt Λɺ

(8)βd,t =sof tmax(Wc · ct ) · δd ,%Ωt =βk,t · γk,t ,P (y|x) Λಉ࣌ʹֶश͢Δɻα1,t,k ͸‫ط‬ଘͷߏจղੳ‫ث‬ͷ݁ՌΛ༻ֶ͍ͯश͞ΕΔɻ‫܇‬࿅σʔλதͷґଘߏ଄໦ʹ͓͍ͯɺ୯‫ ޠ‬wt ͷ਌͕ wj Ͱ͋Δ͜ͱΛ at,j = 1 ͱఆٛ͠ɺ୯‫ ޠ‬wt
ͷ਌͕ wj Ͱ͸ͳ͍͜ͱΛ at,j = 0 ͱఆٛͨ࣌͠ɺఏҊख๏ͷ໨తؔ਺͸ɺ

−logP (y|x) − λ ·n %m%
i=1 t=1

at,j · logα1,t,j ,(10)ͱఆٛ͞ΕΔɻλ ͸ग़ྗϥϕϧͱґଘߏ଄ղੳͷॏཁ౓Λௐ੔͢ΔͨΊͷϋΠύʔύϥϝʔλͰ͋Δɻ

4 ࣮‫ݧ‬ධՁʹ͸ Google จѹॖσʔληοτΛ࢖༻ͨ͠ɻςετσʔλ͸ैདྷ‫ͱڀݚ‬ಉ༷ʹධՁηοτͷઌ಄ 1000 จΛ༻͍ͨɻ‫܇‬࿅σʔλʹ͸ެ։͞Ε͍ͯΔ 20 ສจͷσʔληοτΛ࢖༻ͨ͠ɻ‫܇‬࿅σʔλதʹ 10 ճະຬ͔͠ग़‫͍ͳ͠ݱ‬୯‫ޠ‬͸ະ஌‫ͯ͠ͱޠ‬ѻͬͨɻ඼ࢺɺґଘߏ଄ϥϕϧɺґଘߏ଄໦ղʹ͍ͭͯ͸ɺσʔληοτʹ‫·ؚ‬Ε͍ͯΔґଘߏ଄ղੳ‫ث‬ʹΑΔग़ྗΛར༻ͨ͠ɻ࣮‫͠ࡍʹݧ‬ɺఏҊख๏ΛҎԼͷϕʔεϥΠϯͱൺֱͨ͠ɻͳ͓ɺެฏͳൺֱΛߦ͏ͨΊʹɺશͯͷϕʔεϥΠϯʹର͠
ͯୈ 2 અͰઆ໌ͨ͠ೖྗϕΫτϧΛ࢖༻ͨ͠ɻ

• Tagger: ྆ํ޲ LSTM ͱιϑτϚοΫε૚ʹΑΓߏ੒͞Εɺϥϕϧʹର͢Δ఺ਪఆΛߦ͏ख๏ [8]ɻ(9)• Tagger+ILP: ੔਺‫ܭ‬ը໰୊ʹ‫੍ͮ͘ج‬໿্ͰɺTagger ͷग़ྗείΞΛ༻͍ͯ୯‫ޠ‬ͷ࡟আΛܾఆ͢Δख๏ [11]ɻͷΑ͏ʹ‫͢ࢉܭ‬ΔɻWc ͸ιϑτϚοΫε૚ͷॏΈߦྻΛɺd͸࣍਺ͷू߹Λɺct ͸‫ࡏݱ‬ͷจ຺ΛͦΕͧΕද͢ɻͳ͓ɺγ0,t͸θϩϕΫτϧͰ͋Γɺβ0,t ͸ґଘߏ଄໦ͷ৘ใΛ࢖༻͠ͳ͍

• Bi-LSTM: Fillipova Β [2] ͷߏจղੳ݁ՌΛ༻͍ͳ͍ख๏ͷΤϯίʔμΛɺެฏͳൺֱͷͨΊʹ྆ํ޲ LSTM ʹஔ͖‫ͨ͑׵‬ख๏ɻยํ޲ͷͦΕͧΕ LSTM ͷ࠷ऴঢ়ଶͷk∈{0}∪d←− −→→͜ͱΛҙຯ͢Δɻ·ͨɺจ຺ϕΫτϧ ct ͸ ct = [ h 0 , h n , −s t]ͷΑ͏ʹ‫͞ࢉܭ‬ΕΔɻ্‫ه‬ͷΑ͏ʹ‫͞ࢉܭ‬Εͨ Ωt ͸ग़ྗ૚ͷೖྗʹ݁߹͞ΕΔɻ→۩ମతʹ͸ࣜ (1) ͷ dt ͸݁߹‫ޙ‬ͷϕΫτϧ d′t = [ht , Ωt , −s t]ʹஔ͖‫͑׵‬ΒΕΔɻ·ͨσίʔμͷ࣍ͷೖྗʹ༩͑ΒΕΔ dt΋ಉ༷ʹ d′t ʹஔ͖‫͑׵‬ΒΕΔɻ࿨͕σίʔμͷॳ‫ظ‬ঢ়ଶ΁ͱҾ͖‫͕ܧ‬ΕΔɻ

• Bi-LSTM-Dep: Fillipova Β [2] ͷߏจղੳ݁ՌΛ༻͍Δख๏ (LSTM-PAR-PRES) ͷΤϯίʔμΛɺBi-LSTM ͱಉ͘͡ɺ྆ํ޲ LSTM ʹஔ͖‫ͨ͑׵‬ख๏ɻ• Attn: ஫ҙ‫ߏػ‬Λ࣋ͭ Seq2Seq [10] (concat attention)ɻ
• Base: ୈ 2 અͰઆ໌ͨ͠ɺSeq2Seq ʹ‫ͮ͘ج‬ख๏ɻ

3.4 ໨తؔ਺ߏจղੳ‫ޡ‬ΓͷӨ‫ڹ‬Λ௿‫͢ݮ‬ΔͨΊʹɺఏҊख๏͸ 1 ࣍ͷґଘؔ܎Λද͢஫ҙͷ෼෍ α1,t,k ͱɺϥϕϧͷग़ྗ֬཰ैདྷ‫[ ڀݚ‬11] ʹै͍ɺ୯‫ޠ‬ຒΊࠐΈɺLSTM ͷӅΕ૚ɺ஫ҙϕΫτϧΛ 100 ࣍‫ʹݩ‬ઃఆͨ͠ɻશͯͷख๏Ͱ LSTM ͷӅ― 1098 ―Copyright(C) 2018 The Association for Natural Language Processing.
All Rights Reserved.

InputPakistan signed a resolution on Monday to import 1,300 MW of electricity from Kyrgyz Republic and Tajikistan to overcome power shortage in summer season , said an official press release .

GoldPakistan signed a resolution to import 1,300 MW of electricity from Kyrgyz Republic and Tajikistan .

TaggerTagger-ILP
Base

Pakistan signed a resolution to import 1,300 MW of electricity Tajikistan to overcome shortage .
Pakistan signed resolution to import MW said .
Pakistan signed a resolution to import 1,300 MW of electricity .

Prop (d = {1, 2, 4})

Pakistan signed a resolution to import 1,300 MW of electricity from Kyrgyz Republic and Tajikistan .

Input

US whistleblower Bradley Manning , charged with releasing over 700,000 battlefield reports from Iraq andAfghanistan to Wikileaks , received a sentence of 35 years in prison from a military court Wednesday .

GoldBradley Manning received a sentence of 35 years in prison .

TaggerTagger-ILP
Base

Bradley Manning received a sentence of 35 years .
Bradley Manning received a sentence of years .
Bradley Manning received a sentence of 35 years .

Prop (d = {1, 2, 4})

Bradley Manning received a sentence of 35 years in prison .

ද 2: ςετσʔλதʹଘࡏ͢Δैདྷख๏ͱఏҊख๏ͷѹॖจͷྫ.
Ε૚ͷαΠζΛಉ͡஋ʹ͢ΔͨΊʹɺTagger Λ༻͍Δख๏Ͱ
͸ 3 ૚ΛɺσίʔμΛ༻͍Δख๏Ͱ͸ 2 ૚Λ༻͍ͨɻ͜ͷઃఆʹΑΓɺશͯͷख๏͕ 6 ૚ͷ LSTM Λ࣋ͭ͜ͱʹͳΔɻ඼ࢺͱґଘؔ܎ϥϕϧͷຒΊࠐΈʹ͸ 40 ࣍‫ݩ‬Λ࢖༻ͨ͠ɻॏΈͷॳ‫ظ‬஋ʹ͍ͭͯ͸ɺ-1 ͔Β 1 ͷൣғͰҰ༷෼෍͔ΒϥϯμϜʹαϯϓϦϯάͨ͠ [4]ɻ·֤ͨ LSTM ૚ͷೖྗͰ͸υϩοϓΞ΢τΛߦ͍ɺͦͷൺ཰͸ 0.3 ʹઃఆͨ͠ɻ
ֶशʹ͸ Adam [7] Λ࢖༻͠ɺֶश཰ͷॳ‫ظ‬஋͸ 0.001 ʹઃఆͨ͠ɻֶश͸‫܇‬࿅σʔλશମʹର͠ɺ30 ճߦͬͨɻಉֶ࣌शͷͨΊͷ λ ͸ 1.0 ʹઃఆͨ͠ɻϛχόονͷαΠζ͸
16 ʹઃఆ͠ɺϛχόον͝ͱʹ‫ࠩޡ‬Λฏ‫ͨ͠ۉ‬ɻͳ͓ɺϛχόονͷॱং͸ֶश࣌ʹϥϯμϜʹೖΕସ͑ͨɻclipping
threshold ͸ 5.0 ʹઃఆͨ͠ɻ࣮૷ʹ͸ Dynet Λ༻͍ͨɻධՁʹ͸τʔΫϯ୯Ґͷ F ஋ͱɺROUGE-1,2,L Λ࢖༻ͨ͠ɻ·ͨɺద੾ͳ௕ֶ͕͞श͞Ε͍ͯΔ͜ͱΛ֬ೝ͢ΔͨΊʹɺ∆C = γεςϜͷѹॖ཰ − ਖ਼ղͷѹॖ཰ ΛτʔΫϯ୯ҐͰ‫ͯ͠ࢉܭ‬ධՁͨ͠ɻ͞Βʹɺਂ͍ґଘߏ଄໦Λ࣋ͭจʹ͓͚ΔੑೳΛධՁ͢ΔͨΊʹɺґଘߏ଄໦ͷਂ͕͞ςετσʔλશମͰͷฏ‫ۉ‬஋Ͱ͋Δ 8 Ҏ্ͷจ͚ͩͰͷධՁ΋࣮ࢪͨ͠ɻ·ͨɺॳ‫ظ‬஋ʹΑΔ஋ͷ༳ΕΛߟྀ͠ɺશͯͷείΞΛ
5 ճͷࢼߦͷฏ‫ۉ‬஋ͱͨ͠ɻ
ද 1 ʹ࣮‫݁ݧ‬ՌΛࣔ͢ɻఏҊख๏ Prop (d = {1, 2, 4}) ͸ϕʔεϥΠϯख๏ʹൺ΂΄΅શͯͷείΞͰվળ͕‫ݟ‬ΒΕΔɻಛʹ F ஋Ͱ͸ैདྷͷ.82 [2] Λ্ճΔɺ͜ͷσʔληοτʹ͓͚Δ࠷ߴਫ਼౓Λୡ੒͍ͯ͠Δɻ·ͨɺDEPTH Ͱ͸ɺಛʹ
ROUGE είΞʹ͓͍ͯ ALL ΑΓ΋େ͖ͳվળ͕֬ೝ͞ΕΔ͜ͱ͔ΒɺఏҊख๏͸ਂ͍ґଘߏ଄໦Λ࣋ͭจʹରͯ͠༗ޮͰ͋Δ͜ͱ͕ࣔ͞Εͨɻ
ද 2 ʹɺςετσʔλதͷ࣮ࡍͷѹॖจΛࣔ͢ɻग़ྗྫΑΓɺBase ͷग़ྗ͸Մಡੑ͸ߴ͍͕ɺॏཁՕॴ͕ܽམͯ͠͠·͍ͬͯΔ͜ͱ͕෼͔Δɻ·ͨɺTagger ͷग़ྗʹ͍ͭͯ͸ɺจҙ͕อଘ͞Ε͓ͯΒͣɺ·ͨॏཁՕॴ΋ܽམͯ͠͠·͍ͬͯΔɻ͞ΒʹɺTagger-ILP Ͱ͸ɺ༿ϊʔυͷ৘ใ͕༏ઌతʹ࡟আ͞ΕΔͱ͍͏ಛ௃ͷͨΊʹɺ਺ࣈͷܽམ͕ੜ͍ͯ͡Δɻ͜ΕΒͷ݁Ռ͔ΒɺఏҊख๏͸ՄಡੑΛอ࣋ͭͭ͠ɺॏཁՕॴΛѹॖจʹ࢒͢͜ͱ͕ՄೳͰ͋Δ͜ͱ͕෼͔Δɻ5 ݁࿦ຊߘͰ͸ɺਂ͍ґଘߏ଄໦Λ࣋ͭจΛߴਫ਼౓ʹѹॖ͢ΔͨΊʹɺߴ࣍ͷґଘؔ܎ΛߟྀՄೳͳ஫ҙ‫ߏػ‬Λ Seq2Seq ʹ૊ΈࠐΉख๏ΛఏҊͨ͠ɻGoogle จѹॖσʔληοτΛ༻͍࣮ͨ‫ݧ‬ͷ݁ՌɺఏҊख๏͸ैདྷख๏ʹൺͯ͠ߴਫ਼౓ʹจΛѹॖՄೳͰ͋Δ͜ͱ͕ࣔ͞Εͨɻಛʹਂ͍ґଘߏ଄໦Λ࣋ͭจʹ͓͍ͯվળͷ܏޲͸ΑΓ‫ݦ‬ஶͰ͋ͬͨɻҎ্ΑΓߴ࣍ͷґଘؔ܎ΛߟྀՄೳͳ஫ҙ‫ߏػ‬Λ༻͍Δ͜ͱʹΑͬͯɺਂ͍ґଘߏ଄໦Λ࣋ͭจʹ͓͍ͯจѹॖͷੑೳ͕޲্͢Δ͜ͱ͕ࣔ͞Εͨɻࢀߟจ‫ݙ‬[1] Taylor Berg-Kirkpatrick, Dan Gillick, and Dan Klein. Jointly learning to extract
and compress. In Proceedings of the 49th Annual Meeting of the Association forComputational Linguistics: Human Language Technologies, pp. 481–490, Portland,Oregon, USA, June 2011. Association for Computational Linguistics.
[2] Katja Filippova, Enrique Alfonseca, Carlos A. Colmenares, Lukasz Kaiser, and Oriol
Vinyals. Sentence compression by deletion with lstms. In Proceedings of the 2015
Conference on Empirical Methods in Natural Language Processing, pp. 360–368,
Lisbon, Portugal, September 2015. Association for Computational Linguistics.
[3] Katja Filippova and Yasemin Altun. Overcoming the lack of parallel data in sentencecompression. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1481–1491, Seattle, Washington, USA, October 2013.
Association for Computational Linguistics.
[4] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conferenceon Artificial Intelligence and Statistics, pp. 249–256, 2010.
[5] Kazuma Hashimoto and Yoshimasa Tsuruoka. Neural machine translation withsource-side latent graph parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 125–135, Copenhagen, Denmark,September 2017. Association for Computational Linguistics.
[6] Hongyan Jing. Sentence reduction for automatic text summarization. In Proceedings of the Sixth Conference on Applied Natural Language Processing, pp. 310–315,Seattle, Washington, USA, April 2000. Association for Computational Linguistics.
[7] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization.
CoRR, Vol. abs/1412.6980, , 2014.
[8] Sigrid Klerke, Yoav Goldberg, and Anders Søgaard. Improving sentence compression by learning to predict gaze. In Proceedings of the 2016 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1528–1533, San Diego, California, June 2016. Associationfor Computational Linguistics.
[9] Kevin Knight and Daniel Marcu. Statistics-based summarization-step one: Sentencecompression. AAAI/IAAI, Vol. 2000, pp. 703–710, 2000.
[10] Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches toattention-based neural machine translation. In Proceedings of the 2015 Conferenceon Empirical Methods in Natural Language Processing, pp. 1412–1421, Lisbon, Portugal, September 2015. Association for Computational Linguistics.
[11] Liangguo Wang, Jing Jiang, Hai Leong Chieu, Chen Hui Ong, Dandan Song, andLejian Liao. Can syntax help? improving an lstm-based sentence compression modelfor new domains. In Proceedings of the 55th Annual Meeting of the Associationfor Computational Linguistics (Volume 1: Long Papers), pp. 1385–1393, Vancouver,Canada, July 2017. Association for Computational Linguistics.
[12] Xingxing Zhang, Jianpeng Cheng, and Mirella Lapata. Dependency parsing as headselection. In Proceedings of the 15th Conference of the European Chapter of theAssociation for Computational Linguistics: Volume 1, Long Papers, pp. 665–676,
Valencia, Spain, April 2017. Association for Computational Linguistics.

― 1099 ―Copyright(C) 2018 The Association for Natural Language Processing.
All Rights Reserved.