On the Relationship between Neural Machine Translation and Word Alignment

1 INTRODUCTION . Machine translation aims at modeling the semantic equivalence between a pair of source and target sentences ( Koehn , 2009 ) , and word alignment tries to model the semantic equivalence between a pair of source and target words ( Och & Ney , 2003 ) . As a sentence consists of words , word alignment is conceptually related to machine translation and such a relation can be traced back to the birth of statistical machine translation ( SMT ) ( Brown et al. , 1993 ) , where word alignment is the basis of SMT models and its accuracy is generally helpful to improve translation quality ( Koehn et al. , 2003 ; Liu et al. , 2005 ) . In neural machine translation ( NMT ) , it is also important to study word alignment , because word alignment provides potential ways to understanding black-box NMT models and analyzing their translation errors ( Ding et al. , 2017 ) . Unlike SMT , NMT models do not directly depend on word alignment and prior researches implicitly extract word alignment by attention ( Bahdanau et al. , 2014 ; Mi et al. , 2016 ; Liu et al. , 2016 ) . Unfortunately , this method is not general for all NMT models . In particular , to our surprise , it can only obtain word alignment for NMT models with a single attentional layer , but fails for those with multiple attentional layers , which is the standard for state-of-the-art NMT systems such as TRANSFORMER ( Vaswani et al. , 2017 ) . In this paper , we propose two methods to induce word alignment from general NMT models and answer a fundamental question how much word alignment NMT models can learn . The first method explicitly builds a word alignment model between a pair of source and target word representations encoded by NMT models , and then it learns additional parameters for this word alignment model with the supervision from external aligners similar to ( Mi et al. , 2016 ) and ( Liu et al. , 2016 ) . The second method is more intuitive and flexible : it is parameter-free and thus does not need retraining and external aligners . Its key idea is to measure the prediction difference of a target word if a source word is removed , inspired by ( Arras et al. , 2016 ; Zintgraf et al. , 2017 ) . Unlike the first method , the second one only depends on NMT models , and it thereby facilitates better understanding and interpreting NMT models . Experiments on an advanced NMT model show that both methods achieve much better word alignment than the method by attention . However , word alignment obtained by our methods is still worse than statistical word aligners ( Dyer et al. , 2013 ; Och & Ney , 2003 ) . This raises another natural question why advanced NMT models including less word alignment knowledge but deliver better translations than SMT models based on statistical aligners , which was observed in prior researches yet without deep interpretation ( Tu et al. , 2016 ; Liu et al. , 2016 ) . To answer this question , in the spirit of the proposed prediction difference ( PD ) method , we design a criterion to divide target words in the reference of a test sentence into two categories : those target words mostly contributed from source ( CFS ) words and the other target words mostly contributed from target ( CFT ) words in their histories . 1 Our experiments further demonstrate that NMT models capture good word alignment for the first category ( CFS ) words with accuracy comparable to a statistical aligner , while their word alignment for the second category ( CFT ) words is much worse . Thankfully , most target words in the second category ( CFT ) can be easily decoded by NMT models because of the strong language model effects implicitly learned by NMT models . Finally , we exploit the correlation between translation quality and word alignment quality for neural machine translation . We find that there is a weak correlation between translation and word alignment for all target words , yet a strong correlation between translation and word alignment for those target words mostly contributed from source words , after analyzing translations from several NMT systems . This finding suggests that word alignment errors for CFS words are responsible for translation errors in some extent . This paper makes the following contributions : • It proposes two better methods to acquire word alignment from general NMT models . • It explains why NMT models deliver excellent translations no matter their worse word alignment compared to statistical machine translation . • It empirically shows a strong correlation between translation quality and word alignment accuracy for those target words mostly contributed from source words . 2 PRELIMINARIES . 2.1 NEURAL MACHINE TRANSLATION . Given a source sentence x = 〈x1 , · · · , x|x|〉 and a target sentence y = 〈y1 , · · · , y|y|〉 , NMT aims at maximizing the following conditional probabilities : 2 P ( y | x ) = |y|∏ i=1 P ( yi | y < i , x ) = |y|∏ i=1 P ( yi | sLi ) , ( 1 ) where y < i = 〈y1 , . . . , yi−1〉 denotes a prefix of y with length i − 1 , and sLi is the decoding state of yi . Generally , the conditional distribution P ( yi | sLi ) is somehow modeled within an encoderdecoder framework . In encoding stage , the source sentence x are encoded as a sequence of hidden vectors h by an encoder according to specific NMT models , such as a multi-layer encoder consisting of recurrent neural network ( RNN ) , convolutional neural network ( CNN ) , or self-attention layer . In decoding stage , each decoding state sLi is computed by an L-layer decoder as follows : sli = f ( cli , s l−1 i , s l < i , h ) , l ∈ { 1 , . . . , L } , s0i = yi , ( 2 ) where yi is the word embedding of word yi , f is a general function dependent on a specific NMT model , cli is a context vector in l th layer , computed from h and sl < i according to different NMT models . As the dominant models , attentional NMT models define the context vector cli as a weighted sum of h , where the weight αli = g ( sl−1i , s l < i , h ) is defined by a similarity function g. Due to the space limitation , we refer readers to ( Bahdanau et al. , 2014 ) , ( Gehring et al. , 2017 ) and ( Vaswani et al. , 2017 ) for the details on the definitions of f and g. 1For a sentence 〈y1 , y2 , · · · , yi , · · · , y|y|〉 , a history word of yi can be any yk sufficing k < i . 2Throughout this paper , bold font such as x denote a sequence while regular font such as x denote an element which may be a scaler x , vector x or matrixX . 2.2 ALIGNMENT BY ATTENTION . Since the attention weightαli , j measures the similarity between s l−1 i and hj , it has been widely used to evaluate the word alignment between yi and xj ( Bahdanau et al. , 2014 ; Ghader & Monz , 2017 ) . Once an attentional NMT model has been trained , one can easily extract word alignment A from the attention weight α according to the style of maximum a posterior strategy ( MAP ) as follows : Ai , j ( α ) = { 1 j = argmax j′ αi , j′ 0 o/w , ( 3 ) where Ai , j = 1 indicates yi aligns to xj . For NMT models with multiple attentional layers or multiple head attentional layers as in ( Vaswani et al. , 2017 ) , we sum all attention weights with respect to all layers and heads to a single α before MAP in equation 3 . 3 METHODS TO INDUCING WORD ALIGNMENT . Although attention might obtain some word alignment as described in previous section , it is unknown whether NMT models contain more word alignment information than that obtained by attention . In addition , the method using attention is useful to induce word alignment for attentional NMT models , ( more precisely , those models including a single attentional layer as shown in our experiments ) , whereas it is useless for general NMT models . In this section , in order to induce word alignment from general NMT models , we propose two different methods , which are agnostic to specific NMT models . 3.1 ALIGNMENT BY EXPLICIT ALIGNMENT MODEL . Given a source sentence x , a target sentence y , following ( Liu et al. , 2005 ) and ( Taskar et al. , 2005 ) , we explicitly define a word alignment model as follows : P ( xj | yi , y , x ; W ) = exp ( δ ( xj , yi , x , y ; W ) ) ∑m j′=1 exp ( δ ( xj′ , yi , x , y ; W ) ) , ( 4 ) where δ ( xj , yi , x , y ; W ) is a distance function parametrized by W . Ideally , δ is able to include arbitrary features such as IBM model 1 similar to ( Liu et al. , 2005 ) . However , as our goal is not to achieve the best word alignment but to focus on that captured by an NMT model , we only consider these features completely learned in NMT . Hence , we define the δ ( xj , yi , x , y ; W ) = ( xj‖hj ) > W ( yi‖sLi ) , ( 5 ) where xj and yi are word embeddings of xj and yi learned in NMT , hj is the hidden unit of xj in the encoding network and sLi is the hidden unit of yj in the decoding network , ‖ denotes the concatenation of a pair of column vectors of dimension d , andW is a matrix of dimension 2d× 2d . The explicit word alignment model is trained by maximizing the objective function with respect to W : max W ∑ x , y ∑ ∀j , i : Arefij =1 logP ( xj | yi , y , x ; W ) , ( 6 ) where Arefij is the reference alignment between xj and yi for a sentence pair x and y . As the number of elements in W is up to one million ( i.e. , ( 2 × 512 ) 2 ) , it is not feasible to train it using a small dataset with gold alignment . Therefore , following ( Mi et al. , 2016 ; Liu et al. , 2016 ) , we run statistical word aligner such as FAST ALIGN ( Dyer et al. , 2013 ) on a large corpus and then employ resulting word alignment as the silver alignment Aref for training . Note that our goal is to quantify word alignment learned by an NMT model , and thus we only treat W as the parameter to be learned , which differs from the joint training all parameters including those from NMT models as in ( Mi et al. , 2016 ; Liu et al. , 2016 ) . After training , one obtains the optimizedW and then easily infers word alignment for a test sentence pair 〈x , y〉 via the MAP strategy as defined in equation 3 by setting αi , j′ = P ( xj′ | yi , y , x ; W ) . Note that if word embeddings and hidden units learned by NMT models capture enough information for word alignment , the above method can obtain excellent word alignment . However , because the dataset for supervision in training definitely include some data intrinsic word alignment information , it is unclear how much word alignment is only from NMT models . Therefore , we propose the other method which is parameter-free and only dependent on NMT models themselves .

Summary: This paper sets out to build good bilingual word alignments from the information in an NMT system (both Transformer and RNN), where the goal is to match human-generated word-alignments as measured by AER. At least that’s how it starts. They contribute two aligners: one supervised aligner that uses NMT source and target representations as features and is trained on silver data generated by FastAlign, and one interpretability-based aligner that scores the affinity of a source-target word-pair by deleting the source word (replacing its embedding with a 0-vector) and measuring the impact on the probability of the target word. These are both shown to outperform directly extracting alignments from attention matrices by large margins. Despite the supervised aligner getting better AER, the authors proceed to quickly discard it as they dive deep on the interpretability approach, applying it also to target-target word pairs, and drawing somewhat interesting conclusions about two classes of target words: those that depend most of source context and those that depend most on target context.