Title: ON THE RELATIONSHIP BETWEEN NEURAL MACHINE TRANSLATION AND WORD ALIGNMENT

Abstract: Prior researches suggest that attentional neural machine translation (NMT) is able to capture word alignment by attention, however, to our surprise, it almost fails for NMT models with multiple attentional layers except for those with a single layer. This paper introduce two methods to induce word alignment from general neural machine translation models. Experiments verify that both methods obtain much better word alignment than the method by attention. Furthermore, based on one of the proposed method, we design a criterion to divide target words into two categories (i.e. those mostly contributed from source "CFS" words and the other words mostly contributed from target "CFT" words), and analyze word alignment under these two categories in depth. We find that although NMT models are difficult to capture word alignment for CFT words but these words do not sacrifice translation quality significantly, which provides an explanation why NMT is more successful for translation yet worse for word alignment compared to statistical machine translation. We further demonstrate that word alignment errors for CFS words are responsible for translation errors in some extent by measuring the correlation between word alignment and translation for several NMT systems.

1. INTRODUCTION:

Machine translation aims at modeling the semantic equivalence between a pair of source and target sentences (Koehn, 2009) , and word alignment tries to model the semantic equivalence between a pair of source and target words (Och & Ney, 2003) . As a sentence consists of words, word alignment is conceptually related to machine translation and such a relation can be traced back to the birth of statistical machine translation (SMT) (Brown et al., 1993) , where word alignment is the basis of SMT models and its accuracy is generally helpful to improve translation quality (Koehn et al., 2003; Liu et al., 2005) .

In neural machine translation (NMT), it is also important to study word alignment, because word alignment provides potential ways to understanding black-box NMT models and analyzing their translation errors (Ding et al., 2017) . Unlike SMT, NMT models do not directly depend on word alignment and prior researches implicitly extract word alignment by attention (Bahdanau et al., 2014; Mi et al., 2016; Liu et al., 2016) . Unfortunately, this method is not general for all NMT models. In particular, to our surprise, it can only obtain word alignment for NMT models with a single attentional layer, but fails for those with multiple attentional layers, which is the standard for state-of-the-art NMT systems such as TRANSFORMER (Vaswani et al., 2017) .

In this paper, we propose two methods to induce word alignment from general NMT models and answer a fundamental question how much word alignment NMT models can learn. The first method explicitly builds a word alignment model between a pair of source and target word representations encoded by NMT models, and then it learns additional parameters for this word alignment model with the supervision from external aligners similar to (Mi et al., 2016) and (Liu et al., 2016) . The second method is more intuitive and flexible: it is parameter-free and thus does not need retraining and external aligners. Its key idea is to measure the prediction difference of a target word if a source word is removed, inspired by (Arras et al., 2016; Zintgraf et al., 2017) . Unlike the first method, the second one only depends on NMT models, and it thereby facilitates better understanding and interpreting NMT models. Experiments on an advanced NMT model show that both methods achieve much better word alignment than the method by attention. However, word alignment obtained by our methods is still worse than statistical word aligners (Dyer et al., 2013; Och & Ney, 2003) . This raises another natural question why advanced NMT models including less word alignment knowledge but deliver better translations than SMT models based on statistical aligners, which was observed in prior researches yet without deep interpretation (Tu et al., 2016; Liu et al., 2016 ). To answer this question, in the spirit of the proposed prediction difference (PD) method, we design a criterion to divide target words in the reference of a test sentence into two categories: those target words mostly contributed from source (CFS) words and the other target words mostly contributed from target (CFT) words in their histories. 1 Our experiments further demonstrate that NMT models capture good word alignment for the first category (CFS) words with accuracy comparable to a statistical aligner, while their word alignment for the second category (CFT) words is much worse. Thankfully, most target words in the second category (CFT) can be easily decoded by NMT models because of the strong language model effects implicitly learned by NMT models. Finally, we exploit the correlation between translation quality and word alignment quality for neural machine translation. We find that there is a weak correlation between translation and word alignment for all target words, yet a strong correlation between translation and word alignment for those target words mostly contributed from source words, after analyzing translations from several NMT systems. This finding suggests that word alignment errors for CFS words are responsible for translation errors in some extent. This paper makes the following contributions:

• It proposes two better methods to acquire word alignment from general NMT models.

• It explains why NMT models deliver excellent translations no matter their worse word alignment compared to statistical machine translation.

• It empirically shows a strong correlation between translation quality and word alignment accuracy for those target words mostly contributed from source words.

2. PRELIMINARIES:



2.1. NEURAL MACHINE TRANSLATION:

Given a source sentence x = x 1 , • • • , x |x| and a target sentence y = y 1 , • • • , y |y| , NMT aims at maximizing the following conditional probabilities:2 

EQUATION

where y <i = y 1 , . . . , y i-1 denotes a prefix of y with length i -1, and s L i is the decoding state of y i . Generally, the conditional distribution P y i | s L i is somehow modeled within an encoderdecoder framework. In encoding stage, the source sentence x are encoded as a sequence of hidden vectors h by an encoder according to specific NMT models, such as a multi-layer encoder consisting of recurrent neural network (RNN), convolutional neural network (CNN), or self-attention layer. In decoding stage, each decoding state s L i is computed by an L-layer decoder as follows:

EQUATION

where y i is the word embedding of word y i , f is a general function dependent on a specific NMT model, c l i is a context vector in l th layer, computed from h and s l <i according to different NMT models. As the dominant models, attentional NMT models define the context vector c l i as a weighted sum of h, where the weight α l i = g s l-1 i , s l <i , h is defined by a similarity function g. Due to the space limitation, we refer readers to (Bahdanau et al., 2014) , (Gehring et al., 2017) and (Vaswani et al., 2017) for the details on the definitions of f and g.

2.2. ALIGNMENT BY ATTENTION:

Since the attention weight α l i,j measures the similarity between s l-1 i and h j , it has been widely used to evaluate the word alignment between y i and x j (Bahdanau et al., 2014; Ghader & Monz, 2017) . Once an attentional NMT model has been trained, one can easily extract word alignment A from the attention weight α according to the style of maximum a posterior strategy (MAP) as follows:

EQUATION

where A i,j = 1 indicates y i aligns to x j . For NMT models with multiple attentional layers or multiple head attentional layers as in (Vaswani et al., 2017) , we sum all attention weights with respect to all layers and heads to a single α before MAP in equation 3.

3. METHODS TO INDUCING WORD ALIGNMENT:

Although attention might obtain some word alignment as described in previous section, it is unknown whether NMT models contain more word alignment information than that obtained by attention. In addition, the method using attention is useful to induce word alignment for attentional NMT models, (more precisely, those models including a single attentional layer as shown in our experiments), whereas it is useless for general NMT models. In this section, in order to induce word alignment from general NMT models, we propose two different methods, which are agnostic to specific NMT models.

3.1. ALIGNMENT BY EXPLICIT ALIGNMENT MODEL:

Given a source sentence x, a target sentence y, following (Liu et al., 2005) and (Taskar et al., 2005) , we explicitly define a word alignment model as follows:

EQUATION

where δ (x j , y i , x, y; W ) is a distance function parametrized by W . Ideally, δ is able to include arbitrary features such as IBM model 1 similar to (Liu et al., 2005) . However, as our goal is not to achieve the best word alignment but to focus on that captured by an NMT model, we only consider these features completely learned in NMT. Hence, we define the

EQUATION

where x j and y i are word embeddings of x j and y i learned in NMT, h j is the hidden unit of x j in the encoding network and s L i is the hidden unit of y j in the decoding network, denotes the concatenation of a pair of column vectors of dimension d, and W is a matrix of dimension 2d × 2d.

The explicit word alignment model is trained by maximizing the objective function with respect to W : max

EQUATION

where A ref ij is the reference alignment between x j and y i for a sentence pair x and y. As the number of elements in W is up to one million (i.e., (2 × 512) 2 ), it is not feasible to train it using a small dataset with gold alignment. Therefore, following (Mi et al., 2016; Liu et al., 2016) , we run statistical word aligner such as FAST ALIGN (Dyer et al., 2013) on a large corpus and then employ resulting word alignment as the silver alignment A ref for training. Note that our goal is to quantify word alignment learned by an NMT model, and thus we only treat W as the parameter to be learned, which differs from the joint training all parameters including those from NMT models as in (Mi et al., 2016; Liu et al., 2016) .

After training, one obtains the optimized W and then easily infers word alignment for a test sentence pair x, y via the MAP strategy as defined in equation 3 by setting α i,j = P (x j | y i , y, x; W ).

Note that if word embeddings and hidden units learned by NMT models capture enough information for word alignment, the above method can obtain excellent word alignment. However, because the dataset for supervision in training definitely include some data intrinsic word alignment information, it is unclear how much word alignment is only from NMT models. Therefore, we propose the other method which is parameter-free and only dependent on NMT models themselves.

3.2. ALIGNMENT BY PREDICTION DIFFERENCE:

The intuition to this method is that if y i aligns to x j , the relevance between y i and x j should be much higher than that between y i and any other x k with k = j. Therefore, the key to our method is that how to measure the relevance between y i and x j .

Sampling method Zintgraf et al. (2017) propose a principled method to measure the relevance between a pair of tokens in input and output. It is estimated by measuring how the prediction of y i in the output changes if the input token x j is unknown. Formally, the relevance between y i and x j for a given sentence pair x, y is defined as follows:

EQUATION

with

EQUATION

where x (j,x) = x 1 , . . . , x j-1 , x, x j+1 , . . . , x |x| denotes the sequence by replacing x j with x in x, particularly x (j,∅) = x 1 , . . . , x j-1 , x j+1 , . . . , x |x| denotes the sequence by removing x j from x, P (y i | y <i , x) is defined in equation 1 and P x | y <i , x (j,∅) is approximated by the empirical distribution P (x), which can be considered as the 1-gram language model for the source side of the training corpus. Unlike a computer vision task in (Zintgraf et al., 2017) , the size of source vocabulary in NMT is up to 30000 and thus summation over this large vocabulary is challenging in computational efficiency. As a result, we only sample multiple words to approximate the expectation in equation 8 by Monte Carlo (MC) approach.

Deterministic method Inspired by the idea of dropout (Srivastava et al., 2014) , we measure the relevance by disabling the connection between x j and the encoder network in a deterministic way. Formally, R (y i , x j , x, y) is directly defined via dropout effect on x j as follows:

EQUATION

where x (j,0) denotes the sequence by replacing x j with a word whose embedding is a zero vector. In this way, the computation in equation 9 is much faster than the Monte Carlo sampling approach involving multiple samples. It is worth mentioning that equation 9 resembles the Monte Carlo sampling approach with a single sample in calculation, but it is significantly better than MC with a single sample in alignment quality and is very close to MC approach with enough samples, as to be shown in our experiments.

Note that R(y i , x j , x, y) ∈ [-1, 1],

where R(y i , x j , x, y) = 1 means i th target word is totally determined by the j th source word; R(y i , x j , x, y) = -1 means i th target word and j th source word are mutual exclusive; R(y i , x j , x, y) = 0 means j th source word do not affect generating i th target word. In order to obtain word alignment, after collecting R(y i , x j , x, y) for x j , y i , x and y, one can easily infer word alignment via the MAP strategy as defined in equation 3 by setting α i,j = R(y i , x j , x, y).

Remark The above R(y i , x j , x, y) in equation 7 quantifies the relevance between a target word y i and a source word x j . Similarly, one can quantify the relevance between y i and its history word y k as follows:

EQUATION

where R o indicates the relevance between two target words y i and y k with k < i, and P (y i | y <i (k,0) , x) is obtained by disabling the connection between y k and the decoder network, similar to P y i | y <i , x (j,0) . Unlike R(y i , x j , x, y) capturing word alignment information, R o (y i , y k , x, y) is able to capture word allocation in a target sentence and it will be used to answer a fundamental question why NMT models yields better translation yet worse word alignment compared with SMT in section of experiments.

4. EXPERIMENTS:

In this section, we will empirically explore the following questions through experiments:

1. Does attention really capture word alignment for attentional NMT models?

2. How much word alignment do NMT models learn?

3. Why NMT models deliver better translation yet worse word alignment than SMT?

4. What is the relationship between translation quality and word alignment quality?

To answer these questions, we conduct experiments on Chinese-to-English dataset, which includes many reorderings and thereby is challenging for word alignment and translation tasks. Translation quality is evaluated with case-insensitive 4-gram BLEU (Papineni et al., 2002) , implemented by multi-bleu.perl, and alignment performance is evaluated by AER (Mihalcea & Pedersen, 2003; Koehn, 2009) . In the following experiments and analyzes, both BLEU and AER are shown in percentage.

4.1. SETTING:

The training data consists of 1.8M sentence pairs from Chinese-to-English task of NIST2008 Open Machine Translation Campaign with 40.1M Chinese words and 48.3M English words respectively. The development set is chosen as NIST2002, and the test set is NIST2005. To make NMT model capable of open-vocabulary translation, all the datasets are pre-processed by Byte Pair Encoding (BPE) (Sennrich et al., 2015) with 32K merging operations. 3 To measure word alignment quality, we report AER on NIST05 test set, whose reference alignment was manually annotated by experts (Liu et al., 2016) . The experiments are based on an alignment baseline and three strong translation baselines from NMT and SMT:

• FAST ALIGN (Dyer et al., 2013) : a simple, fast, unsupervised word aligner.

• MOSES (Koehn et al., 2007) : an open source phrase based translation system with default configuration, whose translation tables are derived from FAST ALIGN.

• NEMATUS (Sennrich et al., 2017) : an open source neural machine translation implementation of RNN based sequence-to-sequence model.

• TRANSFORMER (Vaswani et al., 2017) : a novel neural network architecture for language understanding implemented by Zhang et al. (2017) .

More details on training these systems are described in Appendix A.

4.2. DOES ATTENTION REALLY CAPTURE ALIGNMENT?:

To our surprise, word alignment by attention is with AER around 83 for TRANSFORMER with six attentional layers: we examined averaged alignment and that from each attentional layer, but any of their alignment is worse.

Since the bilingual corpus intrinsically includes word alignment in some extent, word alignment by attention should be better than the data intrinsic alignment if attention indeed captures alignment. To obtain the data intrinsic word alignment, we calculate point-wise mutual information (PMI) from the bilingual corpus and then infer word alignment for each bilingual sentence by using the MAP strategy as in equation 3. 4 Referring to Table 1 , alignment captured by attention of sixlayer TRANSFORMER (TRANSFORMER-L6) is obviously worse than alignment by PMI. Hence, it is hard to conclude attention from the standard six-layer TRANSFORMER captures word alignment.

As shown in Table 1 , NEMATUS using single attentional layer of RNN is able to capture better word alignment than PMI, so the architecture of multiple attentional layers in TRANSFORMER may be the reason why its attention fails to capture word alignment. Therefore, we reset TRANSFORMER to include one attentional layer in both encoder and decoder and retrain it on the same dataset. We find that the TRANSFORMER with a single attentional layer (TRANSFORMER-L1) can produce much more reasonable alignment than TRANSFORMER-L6, even if its translation quality substantially decreases. In this sense, we believe that the hidden units h j and s l i in equation 2 represent the information of the entire sequence x and y instead of the information particularly emphasizing the words x j and y i , and thereby attention between h j and s l i is not necessary to indicate the word alignment between x j and y i especially for multi-layer attention models.

4.3. HOW MUCH ALIGNMENT CAN NMT LEARN?:

Previous subsection shows alignment by attention may not be a well-defined method to acquiring alignment particularly for multi-layers models, and thus we need better methods to tell how much alignment NMT models can learn. In this subsection, we analyze performances of the two proposed methods, which includes an explicit alignment model and prediction difference based on TRANSFORMER-L6. One might argue that the reason why the explicit model yields excellent alignment is attributed to the supervision from silver alignment dataset. However, with the same amount of supervision, TRANSFORMER-L1 generates much worse alignment than TRANSFORMER-L6 and it is only comparable to PMI (FAST ALIGN) in AER, as shown in Table 3 . This suggests that supervision is Prediction different can be implemented by sampling method or deterministic method. As shown in Table 4 , the alignment performance of sampling method is improving as growing of the sample size, because the accuracy of Monte Carlo approach is dependent on the number of samples. And no matter what sample size is, the variance of AER is always ignorable. The reason might be arg max operation in equation 3 eliminate the fluctuation of probability matrix. Although using large sample size can achieve nice alignment performance, but it will cost too much computation. Fortunately, the deterministic method can also achieve the nice alignment performance with the same computational cost as sample size equals one in sampling method. As a result, we employ deterministic version as the default for prediction difference in this paper.

Although explicit model achieves better word alignment result than prediction difference, it is difficult to interpret and understand neural machine translation through word alignment from explicit model. The main reason is that explicit model relies on an external aligned dataset with guidance from statistical word aligner FAST ALIGN, and thus the characteristic of its alignment result are similar to that of FAST ALIGN, leading to interpretation biased to FAST ALIGN. On the other hand, prediction difference only relies on prediction from a neural model to define the relevance, it has been successfully used to understand and interpret a neural model (Zintgraf et al., 2017) . Therefore, in the rest of this section, we try to understand TRANSFORMER by using prediction difference in the rest of this section.

4.4. DOES ALIGNMENT ERROR AFFECT TRANSLATION?:

It is still unreasonable that TRANSFORMER is good at translation even though messes up with word alignment. In order to reveal this observation, we firstly divide the target words into two categories:

• Contributing from source (CFS): the prediction of a target word can be mostly attributed to the appearance of a source side word.

• Contributing from target (CFT): the prediction of a target word can be mostly attributed to the appearance of a target side word.

Specifically, we employ prediction difference to define CFS and CFT as follows:

CFS:

EQUATION

)

where ∈ [0, 1) is a probability margin between CFS and CFT words. After dividing the target words into two categories of CFS words and CFT words according to the criterion defined above,5 we calculate their percentages and find that the ratio between CFS and CFT is about 7:3. This result suggests that TRANSFORMER employs more information from source side than target side for translation and it is more important for TRANSFORMER to make better use of source side information.

In addition, it is shown in Table 5 that AER of CFS words by prediction difference improves to 33.95 comparable to AER of CFS by FAST ALIGN, even though AER by prediction difference on overall targets is substantially worse than AER by FAST ALIGN. We further compare word alignment from prediction difference and that from FAST ALIGN according to the CFS words or the CFT words and the results are shown in Table 5 . We find that prediction difference captures good alignment for CFS words as FAST ALIGN does, but its alignment for CFT words is much worse than FAST ALIGN. Additionally, we evaluate the effects of translation quality on both the CFS words and the CFT words by translation recall, which measures the percentages of target words in references recalled by TRANSFORMER and MOSES after decoding. As shown in Table 5 , although TRANSFORMER induces worse alignment for the CFT words, it successfully decodes 62% of the CFT words, which is comparable to the percentages of the CFS words. These results demonstrate that TRANSFORMER is able to easily translate the CFT words by using target history words thanks to its strong language model effect, even if it involves noise source information due to inaccurate alignment. Therefore, CFT words might be the reason why TRANSFORMER yields better translation yet worse word alignment compared to SMT models.

4.5. CORRELATION BETWEEN TRANSLATION AND ALIGNMENT:

We train six translation systems on the same dataset: they share the same model architecture but use encoder-decoder layer number ranging from 1 to 6. As there is only a single manually aligned dataset (i.e. NIST2005), the result might be highly dependent on the specific dataset. Inspired by (Koehn, 2004) , we randomly sample 1200 datasets with replacement from NIST2005, each of which includes the same number of the sentences as NIST2005. We report their BLEU points and AER captured by prediction difference on the 1200 sampled datasets. Figure 1 (a) shows the correlation between translation quality and alignment error rate is only about -0.45. This result indicates a weak correlation between translation and alignment.

Because CFT words yield worse alignment but are easy to translate as observed before, we propose to modify AER on CFS words only. However, since CFS words defined in previous section are dependent on a specific translation model whereas we employ six translation models, there will be a bias for the correlation between translation and alignment on CFS words for different translation models. Hence, we redefine CFS words using two PMI statistics based on a pair of source and target words and a pair of a target word and its history target word, 4 by replacing R and R o with these two PMI statistics in equation 11. It is known that the redefined CFS and CFT are dependent on the training data yet independent on any specific model. In order to avoid redundant concepts, they are still mentioned as CFS and CFT words as before.

In Figure 1 (b), one can see that the modified AER on CFS words correlates well with BLEU points, and such a correlation is up to -0.82. Additionally, as shown in Figure 1 (c), the correlation between translation and alignment for CFT words is around zero and even positive. This indicates that correct alignment of CFT words is not necessary to lead to improved translation quality, because these CFT words are easily figured out by the strong language model effects encoded in NMT models. Together with the result shown in Table 5 , improving word alignment for CFS words is potential to improve translation quality for neural machine translation.

5. RELATED WORK:

In NMT, there are many notable researches which mention word alignment captured by attention in some extent. For example, (Bahdanau et al., 2014) is the first work to show word alignment examples by using attention in an NMT model. Tu et al. (2016) quantitatively evaluate word alignment captured by attention and find that its quality is much worse than statistical word aligners. Motived by this finding, Chen et al. (2016) , Mi et al. (2016) and Liu et al. (2016) improve attention with the supervision from silver alignment results obtained by statistical aligners, in the hope that the improved attention leads to better word alignment and translation quality consequently. Despite the close relation between word alignment and attention, Koehn & Knowles (2017) and Ghader & Monz (2017) discuss the differences between word alignment and attention in NMT. All these works study word alignment for the same kind of NMT models, i.e. that with a single attentional layer. One of our contribution is that we propose model-agnostic methods to study word alignment in a general way which deliver better word alignment quality than attention method. Moreover, for the first time, we further explain and quantify the relationship between translation quality and word alignment for an NMT model.

The prediction difference method in this paper actually provides an avenue to understand and interpret neural machine translation models. Therefore, it is closely related to many works on visualizing and interpreting neural networks (Lei et al., 2016; Bach et al., 2015; Zintgraf et al., 2017) . Indeed, our method is inherited from (Zintgraf et al., 2017) , and our advantage is that it is computationally efficient particularly for those tasks with a large vocabulary. In sequence-to-sequence tasks, Ding et al. (2017) focus on model interpretability by modeling how influence propagates across hidden units in networks, which is often too restrictive and challenging to achieve as argued by (Alvarez-Melis & Jaakkola, 2017) . Instead, Alvarez-Melis & Jaakkola (2017) concentrate on prediction interpretability with only oracle access to the model generating the prediction. To achieve this effect, they propose a casual learning framework to measure the relevance between a pair of source and target words. Our method belongs to the type of prediction interpretability similar to (Alvarez-Melis & Jaakkola, 2017) , but ours is a unified and parameter-free method rather than a pipeline and parameter-dependent one. In addition, both Ding et al. (2017) and Alvarez-Melis & Jaakkola (2017) qualitatively demonstrate interpretability by showing some sentences, while we exhibit the interpretability by quantitatively analyzing all sentences in a test set.

6. CONCLUSION AND FUTURE WORK:

This paper points out that attention is insufficient to induce word alignment or even surprisingly fails for general NMT models. Therefore, it proposes two better methods to induce word alignment than the method by attention for general NMT models. Based on one of the proposed method, it divides target words into two categories including CFS and CFT, and shows that NMT models yield excellent word alignment on CFS words but worse word alignment on CFT words, which do not significantly sacrifice translation quality. This explains a fundamental question why NMT delivers better translation yet worse word alignment than its SMT counterpart. Finally, this paper empirically demonstrates that word alignment errors for CFS words are responsible for translation errors in some extent by measuring the correlation between word alignment quality and translation quality. In the future, we believe that more work is interesting to analyze translation errors such as those errors for CFT words. In addition, we will investigate solutions to improving NMT models, in the hope of using source context and target history context in a more robust manner for better predicting CFS and CFT words.

A TRAINING DETAILS:

We implemented the proposed methods to induce word alignment on TRANSFORMER, since it is the most popular NMT model nowadays. For training MOSES, we use all 1.8M sentences from the corpus, and we train a 4-gram language model based on the target side of its training data. For training both NMT models, only the sentences of length up to 256 tokens are used, with no more than 2 15 tokens in a batch. The dimension of both word embeddings and hidden states are 512. Both encoder and decoder have 6 layers by default, and adopt multi-head attention with 8 heads. The beam size for decoding is 4, and the loss function is optimized by Adam (Kingma & Ba, 2014) , where β 1 = 0.9, β 2 = 0.98 and = 10 -9 . Particularly for the explicit alignment model, the alignment reference is produced by FAST ALIGN.

Note that MOSES and NEMATUS achieve BLEU points on NIST2005 test set comparable to those reported in recent works using the similar corpora (Tu et al., 2016; Liu et al., 2016; Zhou et al., 2017) , and TRANSFORMER achieves much higher performance (i.e. 47 BLEU points). This shows that our MT models are well-trained. 

B CASE STUDY:

where C(µ, ν) is a function for counting occurrence of the pair (µ, ν) according to different scenarios, and Z is the normalizer, i.e. the total number of all possible (µ, ν) pairs. In this paper, we define three types of PMI according to different definitions of C(µ, ν) in the three scenarios as follows.

PMI on Bilingual Data In this scenario, a set of bilingual sentences is given. For a given bilingual sentence x, y , C(y i , x j ) is added by one if both y i ∈ y and x j ∈ x.

PMI on Word Aligned Bilingual Data In this scenario, a set of word aligned bilingual sentences is given. That is, for a bilingual sentence, words in its target side may align to words in its source side. For a given word aligned bilingual sentence (x, y), C(y i , x j ) is added by one if y i ∈ y and x j ∈ x and y i aligns to x j .

PMI between a Word and Its History Word on Monolingual Data In this scenario, a set of monolingual sentences is given. For a given monolingual sentence y, C(y k , y i ) is added by one if y k ∈ y and y i ∈ y with k < i.

D DIFFERENT MARGINS FOR DIVIDING CFS AND CFT:

In equation 11, different margins will partition CFS and CFT differently. As growing of the margin , the partition of CFS and CFT becomes more confident. As shown in Table 6 , more confident CFS words can achieve better alignment performance and translation performance. But the translation recall is still similar between CFS and CFT words, despite the big gap of AER between CFS and CFT words. 

:

For a sentence y 1 , y

, • • • , y i , • • • , y |y| ,a history word of y i can be any y k sufficing k < i. 2 Throughout this paper, bold font such as x denote a sequence while regular font such as x denote an element which may be a scaler x, vector x or matrix X.

Throughout this paper, both BLEU and AER are evaluated after restoring BPE. Particularly for restoring BPE before acquiring alignment via MAP strategy in equation 3, α is firstly merged along target tokens by averaging then merged along source tokens by summation.

More details in Appendix C.

Without affecting main conclusions, = 0 in this experiment for covering all words in analysis. Experiments with different margins are in Appendix D.

Figures and Tables:

Figure 1: Pearson correlation r between translation quality and word alignment captured by the prediction difference method for Overall, CFS and CFT target words.

Figure 2: An example of word alignment and translation produced by SMT and NMT systems. Red arrow means wrong alignment and blue arrow means the prediction is attributed to a target word. The word in light font do not align to any source word, while red word means wrong translation.

Word alignment captured by attention in TRANSFORMER and NEMATUS

Table content: <table><tr><td>Methods</td><td colspan="2">AER BLEU</td></tr><tr><td>PMI</td><td>65.66</td><td>None</td></tr><tr><td colspan="3">TRANSFORMER-L6 (attention) 83.07 * 46.95</td></tr><tr><td colspan="3">TRANSFORMER-L1 (attention) 56.49 36.51</td></tr><tr><td>NEMATUS (attention)</td><td colspan="2">50.60 36.82</td></tr></table>

AER of the proposed methods on TRANSFORMER-L6

Table content: <table><tr><td>Methods</td><td>AER</td></tr><tr><td>FAST ALIGN</td><td>36.57</td></tr><tr><td>NEMATUS (attention)</td><td>50.60</td></tr><tr><td colspan="2">TRANSFORMER-L1 (attention) 56.49</td></tr><tr><td>PMI (FAST ALIGN)</td><td>60.18</td></tr><tr><td>Explicit Alignment Model</td><td>38.88</td></tr><tr><td>Prediction Difference</td><td>41.77</td></tr><tr><td colspan="2">Explicit Alignment Model As the explicit model employs the silver alignment dataset from FAST</td></tr><tr><td colspan="2">ALIGN for training its parameters, its final AER includes contributions from both the aligned data</td></tr><tr><td colspan="2">and the model. We analyze the alignment performance from the model itself by comparing with</td></tr><tr><td colspan="2">PMI (FAST ALIGN), which is similar to PMI but calculates co-occurrence of a word pair aligned by</td></tr><tr><td colspan="2">FAST ALIGN in a bilingual sentence. 4 As shown in Table 2, the explicit model outperforms PMI</td></tr><tr><td colspan="2">(FAST ALIGN), which indicates that the six-layer TRANSFORMER indeed learns word alignment</td></tr><tr><td colspan="2">information. In addition, AER by the explicit model is better than attention method over both one</td></tr><tr><td>layer TRANSFORMER and RNN models.</td><td/></tr></table>

Explicit alignment model on different translation modelsPrediction Difference As shown in Table2, prediction difference delivers better word alignment than the data intrinsic word alignment, i.e. PMI (FAST ALIGN), and this result gives a strong indicator that TRANSFORMER with multiple attentional layers indeed induces reasonable alignment even though the alignment is not captured by its attention. In addition, prediction difference is able to capture better word alignment than attention in the state of the art RNN model NEMATUS. However, both explicit model and prediction difference are worse than statistical word aligner FAST ALIGN in terms of AER.

Table content: <table><tr><td/><td>Models</td><td>L1</td><td>L2</td><td>TRANSFORMER L3 L4</td><td>L5</td><td>L6</td><td>PMI (FAST ALIGN)</td></tr><tr><td>AER</td><td colspan="6">EAM * Attention 56.49 76.96 81.23 81.83 87.15 86.87 54.50 47.94 40.47 38.40 38.80 38.88</td><td>60.18</td></tr><tr><td/><td>BLEU</td><td colspan="5">36.51 44.83 45.63 47.19 46.35 46.95</td><td>N/A</td></tr><tr><td colspan="5">* EAM is the abbreviation of "Explicit Alignment Model".</td><td/><td/></tr><tr><td colspan="8">not enough to obtain good alignment and the hidden units learned by a translation model indeed</td></tr><tr><td colspan="4">implicitly capture alignment knowledge.</td><td/><td/><td/></tr></table>

Comparison between sampling and deterministic methods for prediction difference

Table content: <table><tr><td>Methods Sample size</td><td>1</td><td>2</td><td colspan="2">Sampling method 4 20</td><td>50</td><td>Deterministic method</td></tr><tr><td>AER</td><td>44.92</td><td>43.30</td><td>42.42</td><td>41.83</td><td>41.73</td><td>41.77</td></tr><tr><td>Variance</td><td colspan="5">0.004 &lt; 10 -5 &lt; 10 -5 &lt; 10 -5 &lt; 10 -5</td><td>N/A</td></tr></table>

Word alignment quality affects translation quality Methods Target Words * AER Translation Recall † Overall target words, 70.71% target words belong to CFS and 29.29% target words belong to CFT.

Table content: <table><tr><td/><td>Overall</td><td>41.77</td><td>63.81</td></tr><tr><td>PD &amp; TRANSFORMER-L6</td><td>CFS</td><td>33.95</td><td>64.51</td></tr><tr><td/><td>CFT</td><td>63.28</td><td>62.10</td></tr><tr><td/><td>Overall</td><td>36.57</td><td>60.76</td></tr><tr><td>FAST ALIGN &amp; MOSES</td><td>CFS</td><td>31.66</td><td>61.74</td></tr><tr><td/><td>CFT</td><td>50.80</td><td>58.42</td></tr></table>

Word alignment and translation quality under different partitions of CFS and CFT Target Words AER Translation Recall Proportion

Table content: <table><tr><td/><td>CFS</td><td>31.64</td><td>65.54</td><td>65.61%</td></tr><tr><td>10 -4</td><td>CFT</td><td>62.91</td><td>66.39</td><td>24.66%</td></tr><tr><td/><td>CFS</td><td>30.33</td><td>67.82</td><td>60.40%</td></tr><tr><td>10 -3</td><td>CFT</td><td>63.29</td><td>69.40</td><td>22.04%</td></tr><tr><td/><td>CFS</td><td>28.26</td><td>71.56</td><td>51.53%</td></tr><tr><td>10 -2</td><td>CFT</td><td>64.22</td><td>73.76</td><td>17.56%</td></tr><tr><td/><td>CFS</td><td>22.87</td><td>78.59</td><td>34.85%</td></tr><tr><td>10 -1</td><td>CFT</td><td>64.13</td><td>78.33</td><td>10.39%</td></tr></table>

Peer review (iclr2019):
 This paper empirically evaluates whether NMT can predict word alignment. This is done by measuring the alignment error rate between silver-data generated from FastAlign and various methods to extract alignment from NMT. The conclusions are that NMT attention does not predict alignment well, and the proposed method of training an additional alignment extraction model performs better. Unfortunately, I do not appreciate the motivation of the work. Attention is not alignment. Yes, we can try to extract alignment from attention or other parts of the model. But we should really not expect attention to do alignment, because it was simply not designed to do so. So I am not surprised by the first result that AER for attention-based alignment is bad. I am also not suprised by the second result where training an explicit word alignment model (Eq 6) gets better AER, because that is what the model is designed for. I do understand that probing alignments might increase our ability to understand and interpret these NMT models. I also thought the CFS and CFT analysis was interesting. But unfortunately I don*t think the overall problem attacked by this paper is sufficiently well-motivated for a full paper at ICLR. I do think this will be suitable at, for example, a workshop on visualizing/interpreting neural networks in applications. Additional note: Please be more specific whether the AER results in Sec 4.2-4.4 are based on scoring on gold alignments in the NIST2015 data, or silver alignments from FastAlign. The former is fine, but the latter might make the results biased for the explicit alignment model trained on the same alignments.