Venue: NAACL 2018

# Target Foresight based Attention for Neural Machine Translation+
Footnote †: Work done when X. Li interning at Tencent AI Lab. L. Liu is the corresponding author.

Xintong Li\({}^{\dagger}\), Lemao Liu\({}^{\ddagger}\), Zhaopeng Tu\({}^{\ddagger}\), Shuming Shi\({}^{\ddagger}\), Max Meng\({}^{\dagger}\)

###### Abstract

in neural machine translation, an attention model is used to identify the aligned source words for a target word (target foresight word) in order to select translation context, but it does not make use of any information of this target foresight word at all. previous work proposed an approach to improve the attention model by explicitly accessing this target foresight word and demonstrated the substantial gains in alignment task. however, this approach is useless in machine translation task on which the target foresight word is unavailable. in this paper, we propose a new attention model enhanced by the implicit information of target foresight word oriented to both alignment and translation tasks. empirical experiments on chineset-to-english and japanese-to-english datasets show that the proposed attention model delivers significant improvements in terms of both alignment error rate and bleu.

1The Chinese University of Hong Kong

{xtli, qhmeng}@ee.cuhk.edu.hk

\({}^{\ddagger}\)Tencent AI Lab

{redmondliu, zptu, shumingshi}@tencent.com

## 1 Introduction

Since neural machine translation (NMT) was proposed Bahdanau et al. (2014), it has been attracted increasing interests in machine translation community Luong et al. (2015b); Tu et al. (2016); Feng et al. (2016); Cohn et al. (2016). NMT not only yields impressive translation performance in practice, but also has appealing model architecture in essence. Compared with traditional statistical machine translation Koehn et al. (2003); Chiang (2005), one of advantages in NMT is that its architecture combines language model, translation model and alignment between source and target words in a unified manner rather than a pipeline manner, and it thereby has the potential to alleviate the issue of error propagation.

In NMT, the attention mechanism plays an important role. It calculates the alignments of a target word with respect to the source words for translation context selection. Although the source words are always available in inference, the target word, called target foresight word, 1

Figure 1: A running example to motivate the proposed model. (a) The baseline obtains a translation error due to the incorrect attention. (b) With the help of the target foresight information “VBZ”, TFA-NMT is likely to figure out the exact translation as the reference in (c). The light font denotes the target words to be translated in future. Both dashed or solid arrowed lines denote the alignments and solid one denotes the 1-best alignment.



i.e. the first light color word in Figure 1(a), is not known but to be translated at the next time step. Therefore, this may lead to inadequate modeling for attention mechanism Liu et al. (2016); Peter et al. (2017). Regarding to this, Peter et al. (2017) explicitly feed this target word into the attention model, and demonstrate the significant improvements in alignment accuracy. Unfortunately, this approach relies on the premise that the target foresight word is available in advance in its alignment scenario, and thus it can not be used in the translation scenario.

To address this issue, in this paper, we propose a target foresight based attention (TFA) model oriented to both alignment and translation tasks. Its basic idea includes two steps: it firstly designs an auxiliary mechanism to predict some information for the target foresight word which is helpful for alignment; and then it feeds the predicted result into the attention model for translation. For the sake of efficiency, instead of predicting the target foresight word with large vocabulary size, we only predict its partial information, i.e. part-of-speech tag, which is proved to be helpful for word alignment Liu et al. (2005). Figure 1(b) shows the main idea of TFA based on NMT. In order to remit the negative effects due to the prediction errors, we feed the distribution of the prediction result instead of the maximum a posteriori result into the attention model. In addition, since the target foresight words are available during the training, we jointly learn the prediction model for the target foresight words and the translation model in a supervised manner.

This paper makes the following contributions:

* It proposes a novel TFA-NMT for neural machine translation by using an auxiliary mechanism to predict the target foresight word which is subsequently used to enhance the attention model.
* It empirically shows that the proposed TFA-NMT can lead to better alignment accuracy, and achieves significant improvements on both Chinese-to-English and Japanese-to-English translation tasks.

## 2 Background

Given a source sentence \({\bf x}=\left\{x_{1},\ldots,x_{m}\right\}\) with length \(m\) and a target sentence \({\bf y}=\left\{y_{1},\ldots,y_{n}\right\}\) with length \(n\), neural machine translation aims to model the conditional probability \(P\left({\bf y}~{}|~{}{\bf x}\right)\):

\[P\left({\bf y}~{}|~{}{\bf x}\right)=\prod_{i=1}^{n}P\left(y_{i}~{}|~{}{\bf y} _{<i},{\bf x}\right), \tag{1}\]

where \({\bf y}_{<i}=\left\{y_{1},\ldots,y_{i-1}\right\}\) denotes a prefix of \({\bf y}\) with length \(i-1\).

To achieve this, neural machine translation adopts recurrent neural network (RNN) under the encoder-decoder framework Bahdanau et al. (2014). In encoding, an encoder reads the source sentence \({\bf x}\) into a sequence of representation vectors by a bidirectional recurrent neural network. Suppose \(h_{i}\) denotes the representation vector for \(x_{i}\), and let \({\bf h}=\left\{h_{1},\ldots,h_{m}\right\}\). In decoding, a decoder sequentially generates a target word according to \(P\left(y_{i}~{}|~{}{\bf y}_{<i},{\bf x}\right)\) by using another RNN.

In Eq.(1), the distribution \(P\left(y_{i}~{}|~{}{\bf y}_{<i},{\bf x}\right)\) is used to generate \(y_{i}\) as follows:

\[P\left(y_{i}~{}|~{}{\bf y}_{<i},{\bf x}\right)=\text{softmax}\left(\phi\left( y_{i-1},s_{i},c_{i}\right)\right), \tag{2}\]

where \(\phi\) represents a feedforward neural network, \(c_{i}\) is the context vector from \({\bf h}\) to infer \(y_{i}\), and \(s_{i}\) denotes the hidden state at timestamp \(i\) via the decoding RNN represented by \(f\):

\[s_{i}=f\left(s_{i-1},y_{i-1},c_{i}\right). \tag{3}\]

Bahdanau et al. (2014) propose an attention model to define the context \(c_{i}\), inspired by the alignment model in statistical machine translation.

Given the last hidden state \(s_{i-1}\) and the encoding vectors \({\bf h}\), an attention model is based on a distribution consisting of \(\alpha_{ij}\) as follows:

\[\alpha_{ij}=\frac{\exp\left(e_{ij}\right)}{\sum_{k=1}^{m}\exp\left(e_{ik} \right)},\]

where \(e_{ij}\) is computed by a feedforward neural network represented by \(a\):

\[e_{ij}=a\left(s_{i-1},h_{j}\right). \tag{4}\]

The quantity \(\alpha_{ij}\) denotes the possibility of target word \(y_{i}\) aligns to the source word \(x_{j}\) encoded by \(h_{j}\). According to \(\alpha_{ij}\), the context 

vector \(c_{i}\) is defined as the weighted sum of **h**:

\[c_{i}=\sum_{j=1}^{m}\alpha_{ij}h_{j}. \tag{5}\]

In this way, when translating the target word \(y_{i}\), the decoder will pay more attention to its aligned source words with respect to the distribution \(\alpha_{i}=\left\{\alpha_{i1},\cdots,\alpha_{im}\right\}\). Figure 2 shows a slice of the entire architecture for NMT at timestamp \(i\).

Unfortunately, even though the entire translation **y** is available in training, during the inference it is unknown in advance but to be generated sequentially. Specifically, when calculating \(\alpha_{i}\), one can make use of the information only from **x** and \(\textbf{y}_{<i}\) but nothing from \(y_{i}\). Therefore, it is difficult to certainly specify which source words should be aligned to an unknown target word \(y_{i}\). This might lead to the inadequacy of the attention model Liu et al. (2016a); Peter et al. (2017), as explained in Figure 1(a).

## 3 Target Foresight Attention

In order to alleviate the issue of inadequate modeling for attention in NMT, in this section, we propose the target foresight attention for NMT, which foresees some related information of the unknown target foresight word to improve its alignments regarding to source words. The basic idea of the proposed attention model includes two steps as following:

* It firstly introduce a model to predict some information of the target foresight word. (SS3.1)
* It then feeds the predicted result about the foresight target word into the attention as an additional input. (SS3.2)

Therefore, as shown in Figure 1(b), when translating the third word, if the prediction model shows it to be a "VBZ", the attention model is likely to align it to the verb words such as "hui sheng" rather than "ren shu" in the source side, and then the corrected word "rises" will be translated.

### Target Foresight Prediction

Ideally, it is possible to build a model to directly predict the target foresight word itself. In practice, it will be inefficient due to its large vocabulary size. As a result, we instead build a model to predict the partial information of the target foresight word, such as part-of-speech (POS) tag or word cluster, which has limited vocabulary size. In this paper, we use the POS tag as the partial information of a target foresight word because POS tag is helpful to word alignment proved by Liu et al. (2005). Furthermore, predicting a POS tag is easier than a target foresight word, so the predicted result will be more reliable for the downstream application on attention.

Suppose \(u_{i}\) denotes a variable indicating the POS tag of a target foresight word \(y_{i}\). Our aim is to define a prediction model of \(u_{i}\) prior to calculate the attention probability. For simplicity, this prediction model is generally represented as \(\beta_{i}=P(u_{i}~{}|~{}\mathbf{y}_{<i},\mathbf{x})\). We consider three variant prediction models in a coarse-to-fine manner as follows.

#### 3.1.1 Model 1

It is straightforward to define this prediction model directly based on the hidden states of the RNN in decoder by using a neural network. Formally, one can use the following equation:

\[\beta_{i}=P\left(u_{i}~{}|~{}\mathbf{y}_{<i},\mathbf{x}\right)=\text{softmax }\left(\psi(y_{i-1},s_{i-1})\right), \tag{6}\]

where \(\psi\) is implemented by a feedforward neural network. Note that Eq.(6) only depends on the decoding RNN hidden state \(s_{i-1}\) and it is very simple to implementation. Figure 3(a) shows its architecture.

#### 3.1.2 Model 2

Unlike Eq.(6) relying on the same hidden \(s_{i-1}\) as the decoder, we design a specialized RNN

Figure 2: One slice of the architecture of Neural Machine Translation based on a generic attention.

 

to provide a particular hidden state for prediction of \(u_{i}\). This improved prediction model is defined as follows:

\[\beta_{i}=P\left(u_{i}\mid\mathbf{y}_{<i},\mathbf{x}\right)=\text{softmax}\left( \psi(y_{i-1},t_{i})\right), \tag{7}\]

where \(t_{i}\) is the hidden state of the specialized RNN defined by a GRU unit, i.e. \(t_{i}=g(t_{i-1},y_{i-1})\). This prediction model architecture is shown in Figure 3(b).

#### 3.1.3 Model 3

In model 2, the specialized RNN for \(u_{i}\) only cares about the target sentence \(\mathbf{y}\) and ignores the information from the source sentence \(\mathbf{x}\). We define a fine-grained model by taking a context vector \(c^{\prime}_{i}\) from \(\mathbf{x}\) as an additional input:

\[\beta_{i}=P\left(u_{i}\mid\mathbf{y}_{<i},\mathbf{x}\right)=\text{softmax} \left(\psi(y_{i-1},t_{i},c^{\prime}_{i})\right), \tag{8}\]

where \(c^{\prime}_{i}\) is a context vector extracted from \(\mathbf{x}\) in a way similar to \(c_{i}\) in Eq.(5),2 and \(t_{i}=g(t_{i-1},y_{i-1},c^{\prime}_{i})\) is the hidden state of the specialized RNN. The architecture of this model is shown in Figure 3(c).

Footnote 2: In our preliminary experiments, we tried \(c_{i}\), but we found \(c^{\prime}_{i}\) performs better.

### Feeding the Prediction Model

Suppose we have the prediction result \(P(u_{i}\mid\mathbf{y}_{<i},\mathbf{x})\), then we consider to feed it into the attention model. Firstly, it is natural to feed the prediction into attention by using maximum a posteriori (MAP) strategy:

\[e_{ij}=a(s_{i-1},h_{j},z_{i}), \tag{9}\]

where \(a\) is the function for attention similar to Eq.(4) but includes an additional input \(z_{i}\), which is the MAP result of \(P(u_{i}\mid\mathbf{y}_{<i},\mathbf{x})\):

\[z_{i}=\mathbf{z}\big{(}\operatorname*{argmax}_{u_{i}}P\left(u_{i}\mid\mathbf{ y}_{<i},\mathbf{x}\right)\big{)}, \tag{10}\]

where \(\mathbf{z}\) denotes the embeddings of the POS tags of target foresight words, and \(\mathbf{z}(u_{i})\) returns the embedding of a particular POS tag \(u_{i}\).

Note that in Eq.(10) the accuracy of \(P(u_{i}\mid\mathbf{y}_{<i},\mathbf{x})\) is important to the attention model. For example, suppose at timestamp \(i\), the ground-truth POS tag is "NN", but one has \(P(u_{i}=\text{NN}\mid\mathbf{y}_{<i},\mathbf{x})=0.4\) and \(P(u_{i}=\text{VV}\mid\mathbf{y}_{<i},\mathbf{x})=0.41\). In this case, the prediction model selects "VV" as the POS tag of the target foresight word and ignores the ground-truth tag "NN". Then the attention model takes this error signal and may align the target foresight word to a verb word. Subsequently, this might lead to a translation error.

Therefore, we propose another method to integrate the expected embedding of \(u_{i}\) according to \(P(u_{i}\mid\mathbf{y}_{<i},\mathbf{x})\) into attention as follows:

\[z_{i}=\sum_{u_{i}}\mathbf{z}(u_{i})P\left(u_{i}\mid\mathbf{y}_{<i},\mathbf{x} \right). \tag{11}\]

In this way, \(z_{i}\) can take into account all possible POS tags \(u_{i}\) including the ground-truth result.

Until now, we can obtain the entire architecture of the proposed target foresight attention based NMT (TFA-NMT), as shown in Figure 4. Comparing Figure 4 with Figure 2, the only difference is the variable \(z_{i}\), which is

Figure 3: The prediction coarse-to-fine models for target foresight information: (a) Model 1 using only the decoding hidden state \(s_{i-1}\). (b) Model 2 using a hidden state \(t_{i}\) from a specialized RNN. (c) Models using a hidden state from a specialized RNN enhanced by the representation vector \(c^{\prime}_{i}\) of \(\mathbf{x}\) similar to Eq.(5).

 

obtained from Eq.(10-11) and the prediction model as shown in Figure 3.

Note that the proposed TFA-NMT models the target foresight word, which is a future word regarding to the current time step, to conduct attention calculation. In this sense, it employs the idea of modeling future and thus resembles to the work in Zheng et al. (2017). The main difference is that TFA-NMT models the future from the target side whereas Zheng et al. (2017) models the future from the source side. In addition, Weng et al. (2017) imposes a regularization term by using future words during training. Unlike our approach, their approach does not use future words during the inference because these words are unavailable. Anyway, it is possible to put both their approach and our approach together for further improvements.

### Learning and Inference

Suppose a set of training data is denoted by \(\big{\{}\big{<}\mathbf{x}^{k},\mathbf{y}^{k},\mathbf{u}^{k}\big{>}\mid k=1, \cdots,K\big{\}}\). Here \(\mathbf{x}^{k},~{}\mathbf{y}^{k}\) and \(\mathbf{u}^{k}\) denotes a source sentence, a target sentence and a POS tag sequence of \(\mathbf{y}^{k}\), respectively. Then one can jointly train both the translation model for \(\mathbf{y}^{k}\) and the prediction model for \(\mathbf{u}^{k}\) by minimizing the loss function:

\[\ell=-\sum_{k}\sum_{i}\big{(}\log P(y_{i}^{k}\mid\mathbf{y}_{< i}^{k},\mathbf{x}^{k})+\\ \lambda\log P(u_{i}^{k}\mid\mathbf{y}_{<i}^{k},\mathbf{x}^{k}) \big{)}, \tag{12}\]

where \(P(y_{i}^{k}\mid\mathbf{y}_{<i}^{k},\mathbf{x}^{k})\) is the translation model similar to Eq.(2) with target foresight attention, and \(P(u_{i}^{k}\mid\mathbf{y}_{<i}^{k},\mathbf{x}^{k})\) is the target foresight prediction model as defined in Eq.(6-8), respectively. \(\lambda\geq 0\) is a hyper-parameter that balances the preference between the translation model and target foresight prediction model.

According to the training objective, the proposed TFA-NMT resembles to the multi-task learning, since it jointly learns two tasks similar to Evgeniou and Pontil (2004); Luong et al. (2015a). The difference of our approach is obviously: in this work the prediction result of one model is integrated into the other model, while in their works, two models only share some common hidden states.

In inference, we implement two different decoding methods according two different ways to integrate the foresight prediction model into attention as described in SS3.2. For the MAP feeding style, we optimize \(u_{i}\) according to the loss function in Eq.(12) by beam search besides optimizing \(y_{i}\). However, for the expectation feeding style, we maintain the standard beam search algorithm only regarding to the translation model, i.e. by setting \(\lambda=0\).

## 4 Experiments

We conduct experiments on Chinese-to-English and Japanese-to-English translation tasks. The specific analyses are based on Chinese-to-English task, and the generalization ability is shown by Japanese-to-English task. Case-insensitive 4-gram BLEU is used to evaluate translation quality, and the multi-bleu.perl is adopted as its implementation.

### Setup

DataThe training data for Chinese-to-English task consists of 1.8M sentence pairs from NIST2008 Open Machine Campaign, with 40.1M Chinese words and 48.3M English words respectively. The development set is chosen as NIST2002 (878 sentences) and the test sets are NIST2005 (1082 sentences), NIST2006 (1664 sentences), and NIST2008 (1357 sentences).

For Japanese-to-English translation, we adopt the data sets from NTCIR-9 patent translation task Goto et al. (2013). The training data consists of 2.0M sentence pairs with 53.4M Japanese words and 49.3M English words, the development and test sets respectively contain 2000 sentences with a single ref

Figure 4: Neural machine translation with target Foresight attention. \(\beta_{i}\) is derived from Figure 3, \(z_{i}\) is from Eq.(10-11), and other nodes are similar to ones in Figure 2.

 

erence, following [132; Liu et al., 2016b] for further comparison.

ImplementationWe compare the proposed models with two strong baselines from SMT and NMT:

* Moses(Koehn et al., 2007): an open source phrased based translation system with default configuration.
* Nematus(Sennrich et al., 2017): an generic attention based NMT.

We implement the proposed models on top of Nematus. We use Stanford Log-linear Part-Of-Speech Tagger [49] to produce POS tags for the English side. For both Chinese-to-English and Japanese-to-English tasks, we limit the vocabularies to the most frequent 30K words for both sides. All the out-of-vocabulary words are mapped to a spacial token "UNK". Only the sentences of length up to 50 words are used in training, with 80 sentences in a batch. The dimension of word embedding is 620. The dimensions of both feed forward NN and RNN hidden layer are 1000. The beam size for decoding is 12, and the cost function is optimized by Adadelta with hyper-parameters suggested by Zeiler (2012). Particularly for TFA-NMT, the foresight embedding is also 620, and the hyper-parameter \(\lambda\) is 1.

### Impact of Components

We conduct analyses on Chinese-to-English translation task, to investigate the impact of the added components and to figure out their best configuration for further testing in the next subsection.

#### 4.2.1 Model Architectures

Table 1 lists the speeds and performances of the proposed models. Clearly the proposed approach improves the translation quality in all cases, although there are still considerable differences among the proposed variants.

Model ComplexityThe proposed models introduce a few parameters to the NMT baseline system Nematus, which has 105M parameters. The most complex model (i.e., Model3) introduces 27M new parameters, which are small compared with the baseline model. As seen, the proposed models significantly slows down the training speed, which we attribute to the new softmax operation over the foresight tags and more gradient operations associated with the new training objective, i.e., Eq.(12). For decoding, the most complex model reduces speed by around 30%, which is the cost of the proposed approach for improving translation quality.

PerformanceWe measure the performance with BLEU and the result is shown in Table 1. Model1 marginally improves performance by guiding the decoder states to embed information for predicting foresight tags. Model2 achieves further improvement by introducing a new specific hidden layer to explicitly separate the predict function from the decoder states. Model3 achieves the best performance by adopting an independent attention model to attend corresponding source parts for foresight prediction, which may not be the same as the attended source parts for translation. We conduct the significant test using Kevin Gimpel's toolkit [11]. We found that Model1 is not signif

\begin{table}
\begin{tabular}{c|r|r r||l l} \multirow{2}{*}{**Model**} & \multirow{2}{*}{**\# Para.**} & \multicolumn{2}{c||}{**Speed**} & \multicolumn{2}{c}{**Performance**} \\ \cline{3-6}  & & _Train_ & _Decode_ & _BLEU_ & _FPA_ \\ \hline Nematus & 105M & 2858.8 & 86.6 & 38.65 & – \\ +2-Layer & +6M & 2522.5 & 84.1 & 38.57 & – \\ \hline +Model1 & +2M & 1844.9 & 72.0 & 38.83 & 69.03 \\ +Model2 & +12M & 1666.1 & 70.1 & 39.26 & 69.95 \\ +Model3 & +27M & 1485.2 & 59.1 & **40.63** & **71.91** \\ \end{tabular}
\end{table}
Table 1: Speeds and performances of the proposed models. “Speed” is measured in words/second for both training and decoding, and performances are measured in terms of BLEU scores (“BLEU”) and foresight prediction accuracy (“FPA”) on the development set. Higher BLEU and FPA scores denote better performance.

 

icantly better than baseline, but Model2 is significantly better with p\(<\)0.05 and Model3 is significantly better with p\(<\)0.01. Given that simply introducing an additional layer ("+2-Layer") does not produce any improvement on this data, we believe the gain of our model is not only from the more introduced parameters. Besides, we augment the word embedding by concatenating the POS tag embedding, proposed by Sennrich and Haddow (2016), the BLEU is 38.96, which indicating the improvement of our model is not only from the POS tagging. In order to further validate the improvements of variant proposed models, we evaluate the foresight prediction accuracy (FPA) for three proposed prediction models. We found that the fine-grained Model3 achieves the best FPA, indicating a good estimated foresight is very important to obtain the gains in terms of BLEU.

#### 4.2.2 Analysis on Syntactic Categories

In this experiment, we investigate which category of generated words benefit most from the proposed approach in terms of alignments measured by alignment error rate (AER) Och (2003). We carry out experiments on the evaluation dataset from Liu and Sun (2015), which contains 900 manually aligned Chinese-English sentence pairs. Following Luong et al. (2015b), we force-decode both the bilingual sentences including source and reference sentences to obtain the attention matrices, and then we extract one-to-one alignments by picking up the source word with the highest alignment confidence as the hard alignment. As shown in Table 2, the AER improvements are modest for content words such as Noun, Verb, and adjective ("Adj.") words; but there are substantial improvements for function words such as preposition words ("Prep.") and punctuations ("Punc.").

The reason can be explained as follows. The content words are easy to align with AER under 38 as shown in Table 2, and thus it is more difficult to gain over the BASE. On the other hand, as depicted in Table 2, function words are inherently more difficult than content words. These findings satisfy the linguistic intuition: content words tend to be less involved in multiple potential correspondences than function words, and function words tend to be attached to content words, as pointed out by Pianta and Bentivogli (2004). Fortunately, TFA-NMT can predict the POS tag for target foresight word with high confidence and thus it can improve the alignment quality by using of POS tags, which is useful for alignment task Liu et al. (2005).

It is surprising that the AER for "Prep.", "Det." and "Punc." is relatively low especially for Base. The main reason can be explained from the quantities \(y_{i-1}\), \(s_{i}\), and \(c_{i}\) in Eq.(2) as follows. These highly frequent function words are usually easy to be translated by using the history information from \(y_{i-1}\) and \(s_{i}\) even if \(c_{i}\) is not confident enough. For example, it is relatively easy to guess the "comma" by using the history words in language model task, where there are no bilingual information at all. Therefore, during the training, the model tries to adjust the parameters for highly frequent words from \(y_{i-1}\) and \(s_{i}\) while neglecting the attention model.

#### 4.2.3 Foresight Strategies

Table 3 shows the performances of different foresight strategies in both training and de

\begin{table}
\begin{tabular}{r|r|r|r r} \multirow{2}{*}{**Type**} & \multicolumn{2}{c|}{**FPA**} & \multicolumn{2}{c}{**AER**} \\ \cline{3-5}  & \multicolumn{1}{l|}{} & Ours & Base & Ours \\ \hline Noun & 30.13\% & 77.49 & 28.97 & 26.50 \\ Verb & 12.39\% & 71.94 & 37.06 & 33.93 \\ Adj. & 9.43\% & 55.99 & 34.67 & 31.86 \\ \hline
**Prep.** & 14.66\% & 79.40 & **84.04** & **76.95** \\ Dete. & 10.08\% & 72.06 & 80.15 & 76.51 \\
**Punc.** & 8.01\% & 74.89 & **91.74** & **66.51** \\ \hline Others & 15.30\% & 81.22 & 53.64 & 39.11 \\ \hline All & 100\% & 74.87 & 49.67 & 42.56 \\ \end{tabular}
\end{table}
Table 2: Performances on syntactic categories. “Base” denotes “Nematus”, and Ours denotes the proposed model.

\begin{table}
\begin{tabular}{r|c||c c} \multirow{2}{*}{**Train (\(\lambda\))**} & \multirow{2}{*}{**Decode**} & \multirow{2}{*}{**BLEU**} & \multirow{2}{*}{\(\bigtriangledown\)} \\ \hline
1 & Exp & 40.63 & – \\
0 & Exp & 39.36 & -1.27 \\
1 & Map & 40.34 & -0.29 \\ \end{tabular}
\end{table}
Table 3: Effect of foresight supervision signal in training (i.e., \(\lambda\)) and foresight representations in decoding: Exp for expectation and Map for maximum a posteriori.

 

coding. Without an explicit objective to guide the training of foresight prediction model (i.e., \(\lambda=0\)), the performance decreases by 1.27 BLEU points. When feeding the best foresight predicted result to the attention model (i.e., Map), the performance decreases by 0.29 BLEU points. We attribute this to the propagation of prediction errors, which can be alleviated by using a weighted representation of all predicted results (i.e., Exp).

In the following experiments, we use "\(\lambda=1\) and Exp" as the default setting for the final system TFA-NMT.

### Main Results

Chinese-to-English TaskTable 4 shows the translation performances for the Chinese-to-English translation task. As seen, the proposed approach significantly outperforms the baseline system (i.e., Nematus) in all cases, demonstrating the effectiveness and university of our model.

Japanese-to-English TaskTable 5 shows the translation quality of the NMT baseline and our TFA-NMT on Japanese-to-English task. From the table, we can see that our model still achieves a significant improvement of 1.22 and 1.31 BLEU points on the development and test set, respectively. This shows that the proposed approach works well across different language pairs.

## 5 Related Work

Attention model becomes a standard component for many applications due to its ability of dynamically selecting the informative context from sequential representations. For example, Xu et al. (2015) propose an attention based neural network for image caption task and advance the state-of-the-art results; Yin et al. (2015) put the attention structure between a pair of convolution networks for answer selection, paraphrase identification and textual entailment tasks. In the context of machine translation, the idea of attention based neural networks has been pioneered by Bahdanau et al. (2014); Luong et al. (2015b) and achieved impressive results over the traditional statistical machine translation. Since then many research works have been devoted to improve the neural machine translation by enhancing attention models.

Tu et al. (2016) design a coverage vector for the translation history and then integrates it into the attention model. Similarly, Meng et al. (2016) maintain a tag vector to keep track of the attention history and Sankaran et al. (2016) memorize historical alignments and accumulate them as temporal memory to improve the attention model. In addition, Zhang et al. (2017) improve the attention with a gated operator for encoding states and a decoding state, and and Dutil et al. (2017) enhance attention through a planning mechanism. Furthermore, Feng et al. (2016) adopt a recurrent structure for attention to take long-term dependencies into account, Zhou et al. (2017) propose a look-ahead attention by additionally modeling the translation history, and Cohn et al. (2016) incorporate structural biases into attention models. Recently Chen et al. (2017) introduce the syntactic knowledge into attention models. These works are essentially similar to the propose approach, since we introduce auxiliary information from a target foresight word into the attention model. However, there is a significant difference between our approach and their approaches. Our auxiliary information biases to the word to be translated at next timestep while theirs biases to the information available so far at the current timestep, and thereby our approach is orthogonal to theirs.

The works mentioned above improve the attention models by access auxiliary information, and thus they modify the structure of attention models in both inference and learning. In contrast, Mi et al. (2016); Liu et al. (2016b); Chen et al. (2016) maintain the structure of the attention models in inference but utilize some external signals to supervise the outputs of attention models during the learning. They improve the generalization abilities of attention models by use of the external aligners as the signals, which typically yield alignment results accurate enough to guide the learning of attention.

## 6 Conclusion

It has been argued that the traditional attention model in neural machine translation suf 

fers from model inadequacy due to the lack of information from the target foresight word (Peter et al., 2017; Liu et al., 2016a). To address this issue, this paper proposes a new attention model, which can serve for both alignment and translation tasks, by implicitly making use of the target foresight word. Empirical experiments on Chinese-to-English and Japanese-to-English tasks demonstrate that the proposed attention based NMT delivers substantial gains in terms of both BLEU and AER scores.

In future work, it is promising to exploit other target foresight information such as word cluster besides the POS tags in this paper, and it is also interesting to apply this idea on top of other attention models such as the local attention in Luong et al. (2015b).

## References

* Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. _arXiv preprint arXiv:1409.0473_ .
* Chen et al. (2017) Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2017. Syntax-directed attention for neural machine translation. _arXiv preprint arXiv:1711.04231_ .
* Chen et al. (2016) Wenhu Chen, Evgeny Matusov, Shahram Khadivi, and Jan-Thorsten Peter. 2016. Guided alignment training for topic-aware neural machine translation. _arXiv preprint arXiv:1607.01628_ .
* Chiang (2005) David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In _Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics_. Association for Computational Linguistics, pages 263-270.
* Clark et al. (2011) Jonathan H Clark, Chris Dyer, Alon Lavie, and Noah A Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling or optimizer instability. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2_. Association for Computational Linguistics, pages 176-181.
* Cohn et al. (2016) Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vymolova, Kaisheng Yao, Chris Dyer, and Gholamreza Haffari. 2016. Incorporating structural alignment biases into an attentional neural translation model. _arXiv preprint arXiv:1601.01085_ .
* Dutil et al. (2017) Francis Dutil, Caglar Gulcehre, Adam Trischler, and Yoshua Bengio. 2017. Plan, attend, generate: Planning for sequence-to-sequence models. _arXiv preprint arXiv:1711.10462_ .
* Evgeniou and Pontil (2004) Theodoros Evgeniou and Massimiliano Pontil. 2004. Regularized multi-task learning. In _Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining_. ACM, pages 109-117.
* Feng et al. (2016) Shi Feng, Shujie Liu, Nan Yang, Mu Li, Ming Zhou, and Kenny Q Zhu. 2016. Improving attention modeling with implicit distortion and fertility for machine translation. In _COLING_. pages 3082-3092.
* Goto et al. (2013) Isao Goto, Ka-Po Chow, Bin Lu, Eiichiro Sumita, and Benjamin K Tsou. 2013. Overview of the patent machine translation task at the ntcri-10 workshop. In _NTCIR_.
* Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In _Proceedings of the 45th annual meeting

\begin{table}
\begin{tabular}{c|l|c|c c c c}
**System** & **Model** & **Dev** & **MT05** & **MT06** & **MT08** & **Ave.** \\ \hline  & Moses & – & 35.4 & 33.7 & 25.0 & 31.37 \\  & NMT-J & – & 36.8 & 36.9 & 28.5 & 34.07 \\ \hline  & SA-NMT & 40.0 & **37.8** & 37.6 & 29.9 & 35.10 \\ \hline \hline \multirow{2}{*}{_This work_} & Nematus & 38.65 & 36.32 & 36.10 & 28.24 & 33.55 \\  & TFA-NMT & **40.63** & 37.70 & **38.01** & **30.12** & **35.28** \\ \end{tabular}
\end{table}
Table 4: Evaluation of translation performance on Chinese-to-English task.

\begin{table}
\begin{tabular}{c|c|c|c}
**System** & **Model** & **Dev** & **Test** \\ \hline  (Liu et al., 2016b) & Moses & 28.6 & 30.2 \\  & NMT-J & 33.0 & 34.1 \\ \hline \hline \multirow{2}{*}{_This work_} & Nematus & 33.92 & 35.01 \\  & TFA-Nmt & 35.14 & 36.32 \\ \end{tabular}
\end{table}
Table 5: Evaluation of translation performance on Japanese-to-English task.

 

of the ACL on interactive poster and demonstration sessions_. Association for Computational Linguistics, pages 177-180.
* Koehn et al. (2003) Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In _Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1_. Association for Computational Linguistics, pages 48-54.
* Liu et al. (2016a) Lemao Liu, Masao Utiyama, Andrew Finch, and Eiichiro Sumita. 2016a. Neural machine translation with supervised attention. _arXiv preprint arXiv:1609.04186_ .
* Liu et al. (2016b) Lemao Liu, Masao Utiyama, Andrew M Finch, and Eiichiro Sumita. 2016b. Agreement on target-bidirectional neural machine translation. In _HLT-NAACL_. pages 411-416.
* Liu et al. (2005) Yang Liu, Qun Liu, and Shouxun Lin. 2005. Loglinear models for word alignment. In _Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics_. Association for Computational Linguistics, pages 459-466.
* Liu and Sun (2015) Yang Liu and Maosong Sun. 2015. Contrastive unsupervised word alignment with non-local features. In _AAAI_. pages 2295-2301.
* Luong et al. (2015a) Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015a. Multitask sequence to sequence learning. _arXiv preprint arXiv:1511.06114_ .
* Luong et al. (2015b) Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015b. Effective approaches to attention-based neural machine translation. _arXiv preprint arXiv:1508.04025_ .
* Meng et al. (2016) Fandong Meng, Zhengdong Lu, Hang Li, and Qun Liu. 2016. Interactive attention for neural machine translation. _arXiv preprint arXiv:1610.05011_ .
* Mi et al. (2016) Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. 2016. Supervised attentions for neural machine translation. _arXiv preprint arXiv:1608.00112_ .
* Och (2003) Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In _Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1_. Association for Computational Linguistics, pages 160-167.
* Peter et al. (2017) Jan-Thorsten Peter, Arne Nix, and Hermann Ney. 2017. Generating alignments using target foresight in attention-based neural machine translation. _The Prague Bulletin of Mathematical Linguistics_ 108(1):27-36.
* Pianta and Bentivolgi (2004) Emanuele Pianta and Luisa Bentivolgi. 2004. Knowledge intensive word alignment with knowns. In _Proceedings of the 20th international conference on Computational Linguistics_. Association for Computational Linguistics, page 1086.
* Sankaran et al. (2016) Baskaran Sankaran, Haitao Mi, Yaser Al-Onaizan, and Abe Itstcyheriah. 2016. Temporal attention model for neural machine translation. _arXiv preprint arXiv:1608.02927_ .
* Sennrich et al. (2017) Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hirschler, Marcin Junczys-Dowmunt, Samuel Laubli, Antonio Valerio Miceli Barone, Jozef Mokry, et al. 2017. Nematus: a toolkit for neural machine translation. _arXiv preprint arXiv:1703.04357_ .
* Sennrich and Haddow (2016) Rico Sennrich and Barry Haddow. 2016. Linguistic input features improve neural machine translation. _arXiv preprint arXiv:1606.02892_ .
* Toutanova et al. (2003) Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In _Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1_. Association for Computational Linguistics, pages 173-180.
* Tu et al. (2016) Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. _arXiv preprint arXiv:1601.04811_ .
* Weng et al. (2017) Rongxiang Weng, Shujian Huang, Zaixiang Zheng, XIN-YU DAI, and Jiajun CHEN. 2017. Neural machine translation with word predictions. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Copenhagen, Denmark, pages 136-145.
* Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In _International Conference on Machine Learning_. pages 2048-2057.
* Yin et al. (2015) Wenpeng Yin, Hinrich Schutze, Bing Xiang, and Bowen Zhou. 2015. Abcnn: Attention-based convolutional neural network for modeling sentence pairs. _arXiv preprint arXiv:1512.05193_ .
* Zeiler (2012) Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. _arXiv preprint arXiv:1212.5701_ .
* Zhang et al. (2017) Biao Zhang, Deyi Xiong, and Jinsong Su. 2017. A gru-gated attention model for neural machine translation. _arXiv preprint arXiv:1704.08430_ 

Zaixiang Zheng, Hao Zhou, Shujian Huang, Lili Mou, Xinyu Dai, Jiajun Chen, and Zhaopeng Tu. 2017. Modeling past and future for neural machine translation. _arXiv preprint arXiv:1711.09502_ .
* Zhou et al. (2017) Long Zhou, Jiajun Zhang, and Chengqing Zong. 2017. Look-ahead attention for generation in neural machine translation. _arXiv preprint arXiv:1708.09217_ .