Chunk-based Decoder for Neural Machine Translation
Shonosuke Ishiwatari¹, Jingtao Yao², Shujie Liu³, Mu Li³, Ming Zhou³, Naoki Yoshinaga⁴, Masaru Kitsuregawa⁴⁵, Weijia Jia²
¹The University of Tokyo²Shanghai Jiao Tong University
⁴Institute of Industrial Science, the University of Tokyo

Overview

Translation between distantlanguage pairs are diﬃcult
1. Some languages use manywords to represent one thing
while others use less words³Microsoft Research Asia
⁵National Institute of Informatics

Idea: Using a chunkrather than a word asbasic translation unit

Our decoder for NMT
decodes sentences in a“chunk-by-chunk” manner toovercome the diﬀerencesof length and word-order

Sequence of a sentence
becomes much shorter
Fixed word order andfree chunk order can bemodeled independenlty

2. Some languages are free
word-order while others are not

Diﬃculties of translation of En -> Ja

SOTA performance indistant languages (En -> Ja)

Word-based Encoder-Decoder

Japanese sentence has

Enc-Dec with Attention [Bahdanau+ 15]- longer sequence (En: 25 vs. Ja: 30 [words/sentence])- free chunk order (e.g.,「だれかが / 犬に」=「犬に / だれかが」)- encodes / decodes “word-by-word”
0'1+-'2

かだれが犬…

!""

I heard that someone was bitten by a dog,weren’ t you injured?

＋/&&'(&%+(だれ か が / 犬 に / 噛ま れ た そう だ けれど、/
君 は / 怪我 し なかっ た ？

3(1+-'2* α(4, dog)!""*+,'+('!"#$%&&'($)"-+.

Proposal: Chunk-based Decoder for Neural Machine Translation
Two additional connections

Two step decoding

1. to capture the interaction between chunks
2. to memorize peprevious outputs well

- First a chunk, then words inside the chunk

9.6('*?

-"./0(/*1(2#($3(14だれかがend-ofchunk

9.6('1*:;<(6*=./6*./6(/*8$6*:/((*=./6*./6(/*;$6(7($6($0'>4
犬

だれにかがend-ofchunk

1.0('*23*4$5(/&6"#$%*6.$$(657.$
犬に-./0&'()('*+,,5./6&'()('*+,,だれかが!"#$%&'()('*+,,!"#$%&'()('*+,,

犬に#7680(

1.0('*83*-./0&5.&6"#$%*9((0:;6%Experiments

Translation examples

Data- ASPEC [Nakazawa+ 16], 1.6M En/Ja pairs

Preprocessing- Bunsetsu chunking with J.DepP [Yoshinaga & Kitsuregawa 09]!"#$%&!"#$%!&'()"#*+,$-.'/$
0+(,"%*-#$+( %$"'&'-#1#'2($#.#,!*+,$0+(,"%*-#$'3!%+1#0$34$%)).4+1-$%.!#*1%!+1-$5'.!%-#$%6!#*$+1!*'02,+1-$%!&'()"#*+,$
7#$-%($+1$%$!4)+,%.$0+#.#,!*+,$
3%**+#*$0+(,"%*-#$*#%,!'*$8

3#,%2(#$2(#*$')#*%!+'1$+($+&)'*!%1!$6'*$!"#$+0#%$(2))'*! +1$&%!#*+%.$0#5#.')&#1!$9$%1$+1!#*6%,#$6'*$%$(23(!%1,#$')#*%!+'1$%!$%!'&+,$.#5#.$/%($0#5#.')#0$8$'&(&$&)%&

   WO 9 VB < )/ MB / %(  3? S +/"-  L7  FK BS  T@  D   R5  MB 

QI ;6        .& H  UE A>    =: 1,0  CN UE    !2
(*#!'  ;6  WO 9  VB < )/ MBGJ P 
 3? 4  $'  L7   8 FK  T@        R5 MB   QI ;6     .& UE A>   =: 1,0   CNUE    !2( *#!' ;6     Baseline systems

1. Word-based encoder-decoder [Bahdanau+ 15]
2. Tree-based encoder [Eriguchi+ 16] (SOTA)

Results

Quality of translation
8$%"98$%"9'F<.=+&'()*+0*$?.+*=-.+*/ 3@0:;<=>?@AB"%'%"#$%"&'CD&$*$B"%E-.+*/08+,-.

/0+-1

FG3742737F

17568

6853:

1954;

Quality ofgenerated chunks!"#$%"&'()*"+,-.

/0+-1!"#$%&'()*+,-.+*/012

2345 67327

685;8!"#$%&'()*+,-.+*/032

4546 78596!"#$%&'()*+,-.+*/082

457: 7;594

A=**&'()*+0*$?.+*=0BC=DE#?"D@08>F0@0<.=+&'()*+0+*?.+*=,195:82,685>>2<.=+&'()*+0*$?.+*=&+*?.+*=0BG("+($(#@087F

1>511,195>9268533,685>;2<.=+&'()*+457> 7;541※Scores inside () are not our implementations*$");4%2#)94<"%.=>")  ?/#::"$=@4>/4A>//>);*"$+,-./&+01.2+.).#34567

82#)9,-./&+0:$":"/&+7    :$W QI ;6  :$
 :$$$$$
O 9  :$VB < )/ MB     :$.GJ P  :$3?  $'  :$ & UE :$A>   L7   :$8  :$FK  :$T   :$=: 1,0   :$@   :$D   :$R5  :$ CN UE  :$  :$!2MB   ( *#!'  :$;6   