Venue: NAACL 2016

# Incorporating Structural Alignment Biases into an Attentional Neural Translation Model

 Trevor Cohn and Cong Duy Vu Hoang&Ekaterina Vymolova University of Melbourne Melbourne, VIC, Australia tcohn@unimelb.edu.au &cwhoang2,evylomova\(\dagger\)student.unimelb.edu.au &Kaisheng Yao Microsoft Research Redmond, WA, USA kaisheng.YAO@microsoft.com &Chris Dyer Carnegie Mellon University Pittsburgh, PA, USA cdyer@cmu.edugholamreza.haffari@monash.edu

###### Abstract

Neural encoder-decoder models of machine translation have achieved impressive results, rivalling traditional translation models. However their modelling formulation is overly simplistic, and omits several key inductive biases built into traditional models. In this paper we extend the attentional neural translation model to include structural biases from word based alignment models, including positional bias, Markov conditioning, fertility and agreement over translation directions. We show improvements over a baseline attentional model and standard phrase-based model over several language pairs, evaluating on difficult languages in a low resource setting.

## 1 Introduction

Recently, models of end-to-end machine translation based on neural network classification have been shown to produce excellent translations, rivalling or in some cases surpassing traditional statistical machine translation systems [9, 30, 4]. This is despite the neural approaches using an overall simpler model, with fewer assumptions about the learning and prediction problem.

Broadly, neural approaches are based around the notion of an _encoder-decoder_[35], in which the source language is _encoded_ into a distributed representation, followed by a _decoding_ step which generates the target translation. We focus on the _attentional model_ of translation [6] which uses a dynamic representation of the source sentence while allowing the decoder to _attend_ to different parts of the source as it generates the target sentence. The attentional model raises intriguing opportunities, given the correspondence between the notions of attention and alignment in traditional word-based machine translation models [7].

In this paper we map modelling biases from word based translation models into the attentional model, such that known linguistic elements of translation can be better captured. We incorporate _absolute positional bias_ whereby word order tends to be similar between the source sentence and its translation (e.g., IBM Model 2 and [8]), _fertility_ whereby each instance of a source word type tends to be translated into a consistent number of target tokens (e.g., IBM Models 3, 4, 5), _relative position bias_ whereby prior preferences for monotonic alignments/attention can be encouraged (e.g., IBM Model 4, 5 and HMM-based Alignment [21]), and _alignment consistency_ whereby the attention in _both_ translation directions are encouraged to agree (e.g. symmetrisation heuristics [11] or joint modelling [12, 1]).

We provide an empirical analysis of incorporating the above structural biases into the attentional model, considering low resource translation scenario over four language-pairs. Our results demonstrate consistent improvements over vanilla encoder

decoder and attentional model in terms of the perplexity and BLEU score, e.g. up to 3.5 BLEU points when re-ranking the candidate translations generated by a state-of-the-art phrase based model.

## 2 The attentional model of translation

We start by reviewing the attentional model of translation Bahdanau et al. (2015), as illustrated in Fig. 1, before presenting our extensions in SS3.

EncoderThe encoding of the source sentence is formulated using a pair of RNNs (denoted _bi-RNN_) one operating left-to-right over the input sequence and another operating right-to-left,

\[{\mathbf{h}^{\rightarrow}_{i}}=\text{RNN}({\mathbf{h}^{\rightarrow}_{i-1}},{\mathbf{r} ^{(\mathrm{s})}_{s_{i}}})\]

where \({\mathbf{h}^{\leftarrow}_{i}}\) and \({\mathbf{h}^{\leftarrow}_{i}}\) are the RNN hidden states. The left-to-right RNN function is defined as

\[{\mathbf{h}^{\rightarrow}_{i}}=\tanh\Big{(}{\mathbf{W}^{\rightarrow}_{si}}{\mathbf{r} ^{(\mathrm{s})}_{s_{i}}}+{\mathbf{W}^{\rightarrow}_{sh}}{\mathbf{h}^{\rightarrow}_{ i-1}}+{\mathbf{b}^{\rightarrow}_{s}}\Big{)} \tag{1}\]

where \({\mathbf{h}^{\rightarrow}_{0}}\in\mathbb{R}^{H}\) is a learned parameter vector, as are \(\mathbf{R}^{(\mathrm{s})}\in\mathbb{R}^{V_{S}\times E}\), \({\mathbf{W}^{\rightarrow}_{si}}\in\mathbb{R}^{H\times E}\), \({\mathbf{W}^{\rightarrow}_{sb}}\in\mathbb{R}^{H\times H}\) and \({\mathbf{b}^{\rightarrow}_{s}}\in\mathbb{R}^{H}\), with \(H\) the number of hidden units, \(V_{S}\) the size of the source vocabulary and \(E\) the word embedding dimensionality.1 Each source word is then represented as a pair of hidden states, one from each RNN, \({\mathbf{e}_{i}}=\bigg{[}\begin{array}{c}{\mathbf{h}^{\rightarrow}_{i}}\\ {\mathbf{h}^{\leftarrow}_{i}}\end{array}\bigg{]}\). This encodes not only the word but also its left and right context, which can provide important evidence for its translation.

Footnote 1: Similarly, \({\mathbf{h}^{\leftarrow}_{0}}\in\mathbb{R}^{H},{\mathbf{W}^{\leftarrow}_{si}}\in \mathbb{R}^{H\times E},{\mathbf{W}^{\leftarrow}_{sh}}\in\mathbb{R}^{H\times H},{ \mathbf{b}^{\rightarrow}_{s}}\in\mathbb{R}^{H}\) are the parameters of the right-to-left RNN. Note that we use a long short term memory unit Hochreiter and Schmidhuber (1997) in place of the RNN, shown here for simplicity of exposition.

A crucial question is how this dynamic sized matrix \(\mathbf{E}=[{\mathbf{e}_{1}},{\mathbf{e}_{2}},\ldots,{\mathbf{e}_{I}}]\in\mathbb{R}^{I \times H}\) can be used in the decoder to generate the target sentence. As with Sutskever's encoder-decoder, the target sentence is created left-to-right using an RNN, while the encoded source is used to bias the process as an auxiliary input. The mechanism for this bias is by attentional vectors, i.e. vectors of scores over each source sentence location, which are used to aggregate the dynamic source encoding into a fixed length vector.

DecoderThe decoder operates as a standard RNN over the translation \(t\), formulated as follows

\[{\mathbf{g}_{j}} =\tanh\Big{(}{\bf W}^{(\mathsf{th})}{\mathbf{g}_{j-1}}+{\bf W}^{( \mathsf{ti})}{\mathbf{r}^{(\mathrm{l})}_{t_{j-1}}}+{\bf W}^{(\mathsf{ta})}{\mathbf{c}_{ j}}\Big{)} \tag{2}\] \[{\mathbf{u}_{j}} =\tanh\Big{(}{\mathbf{g}_{j}}+{\bf W}^{(\mathsf{uc})}{\mathbf{c}_{j}}+{ \bf W}^{(\mathsf{ui})}{\mathbf{r}^{(\mathrm{i})}_{t_{j-1}}}\Big{)}\] (3) \[t_{j} \sim\operatorname*{softmax}\Big{(}{\bf W}^{(\mathsf{ou})}{\mathbf{ u}_{j}}+{\mathbf{b}^{(\mathsf{uo})}}\Big{)} \tag{4}\]

where the decoder RNN is defined analogously to Eq 1 but with an additional input, the source attention component \({\mathbf{c}_{j}}\in\mathbb{R}^{2H}\) and weighting matrix \({\bf W}^{(\mathsf{ua})}\in\mathbb{R}^{H\times 2H}\). The hidden state of the recurrence is then passed through a single hidden layer2 (Eq 3) in combination with the source attention and target word using weighting matrices \({\bf W}^{(\mathsf{uc})}\in\mathbb{R}^{H\times 2H}\) and \({\bf W}^{(\mathsf{ui})}\in\mathbb{R}^{H\times E}\). In Eq 4 this vector is transformed to be target vocabulary sized, using weight matrix \({\bf W}^{(\mathsf{ou})}\in\mathbb{R}^{V_{T}\times H}\) and bias \({\mathbf{b}^{(\mathsf{to})}}\in\mathbb{R}^{V_{T}}\), after which a \(\operatorname*{softmax}\) is taken, and the resulting normalised vector used as the parameters of a Categorical distribution in generating the next target word.

Footnote 2: In Bahdanau et al. (2015) they use a max-out layer for this final step, however we found this to be a needless complication, and instead use a standard hidden layer with tanh activation.

The presentation above assumes a simple RNN is used to define the recurrence over hidden states, however we can easily use alternative formulations of recurrent networks including multiple-layer RNNs, gated recurrent units GRU; Cho et al. (2014)), or long short-term memory LSTM; Hochreiter and Schmidhuber (1997)) units. These more advanced methods allow for more efficient learning of more complex concepts, particularly

Figure 1: Attentional model of translation Bahdanau et al. (2015). The encoder is shown below the decoder, and the edges connecting the two corresponding to the attention mechanism. Heavy edges denote a higher attention weight, and these values are also displayed in matrix form, with one row for each target word.

 

long distance effects. Empirically we found LSTMs to be the best performing, and therefore use these units herein.

The last key detail is the attentional component \({\mathbf{c}_{j}}\) in Eqs 2 and 3, which is defined as follows

\[f_{ji} ={\mathbf{v}^{\top}}\tanh\left({\bf W}^{\text{(ae)}}{\mathbf{e}_{i}}+{\bf W }^{\text{(ah)}}{\mathbf{g}_{j-1}}\right) \tag{5}\] \[{\mathbf{\alpha}_{j}} =\operatorname*{softmax}\left({\mathbf{f}_{j}}\right)\] \[{\mathbf{c}_{j}} =\sum_{i}\alpha_{ji}{\mathbf{e}_{i}}\]

with the scalars \(f_{ji}\) denoting the compatibility between the target hidden state \({\mathbf{g}_{j-1}}\) and the source encoding \({\mathbf{e}_{i}}\). This is defined as a neural network with one hidden layer of size \(A\) and a single output, parameterised by \({\bf W}^{\text{(ae)}}\in\mathbb{R}^{A\times 2H}\), \({\bf W}^{\text{(ah)}}\in\mathbb{R}^{A\times H}\) and \({\mathbf{v}}\in\mathbb{R}^{A}\). The \(\operatorname*{softmax}\) then normalises the scalar compatibility values such that for a given target word \(j\), the values of \(\alpha_{j}\) can be interpreted as alignment probabilities to each source location. Finally, these alignments are used to to reweight the source components \(E\) to produce a fixed length context representation.

Training of this model is done by minimising the cross-entropy of the target sentence, measured word-by-word as for a language model. We use standard stochastic gradient optimisation using the back-propagation technique for computation of partial derivatives according to the chain rule.

## 3 Incorporating Structural Biases

The attentional model, as described above, provides a powerful and elegant model of translation in which alignments between source and target words are learned through the implicit conditioning context afforded by the attention mechanism. Despite its elegance, the attentional model omits several key components of a traditional alignment models such as the IBM models Brown et al. (1993) and Vogel's hidden Markov Model Vogel et al. (1996) as implemented in the GIZ++ toolkit Och and Ney (2003). Combining the strengths of this highly successful body of research into a neural model of machine translation holds potential to further improve modelling accuracy of neural techniques. Below we outline methods for incorporating these factors as structural biases into the attentional model.

### Position bias

First we consider position bias, based on the observation that a word at a given relative position in the source tends to align to a word at a similar relative position in the target, \(\frac{i}{I}\approx\frac{j}{J}\)Dyer et al. (2013). Related, the IBM model 2 learns discrete mappings between positions \(i\) and \(j\) conditioned on sentence lengths \(I\) and \(J\).

We include a position bias through redefining the pre-normalised attention scalars \(f_{ji}\) in Eq 5 as:

\[f_{ji} ={\mathbf{v}^{\top}}\tanh\big{(}{\bf W}^{\text{(ae)}}{\mathbf{e}_{i}}+{ \bf W}^{\text{(ah)}}{\mathbf{g}_{j-1}}+\] \[{\bf W}^{\text{(ap)}}\psi(j,i,I)\big{)} \tag{6}\]

where the new component in the input is a simple feature function of the positions in the source and target sentences and the source length,

\[\psi(j,i,I)=\bigg{[}\log(1+j),\log(1+i),\log(1+I)\bigg{]}^{\top}\]

and \({\bf W}^{\text{(ap)}}\in\mathbb{R}^{A\times 3}\). We exclude the target length \(J\) as this is unknown during decoding, as a partial translation can have several (infinite) different lengths. The use of the \(\log(1+\cdot)\) function is to avoid numerical instabilities from widely varying sentence lengths. The non-linearity in Eq 6 allows for complex functions of these inputs to be learned, such as relative positions and approximate distance from the diagonal, as well as their interactions with the other inputs (e.g., to learn that some words are exceptional cases where a diagonal bias should not apply).

### Markov condition

The HMM model of translation Vogel et al. (1996) is based on a Markov condition over alignment random variables, to allow the model to learn local effects such as when \(i\gets j\) is aligned then it is likely that \(i+1\gets j+1\) or \(i\gets j+1\). These correspond to local diagonal alignments or one-to-many alignments, respectively. In general, there are many correlations between the alignments of a word and the alignments of the preceding word.

Markov conditioning can also be incorporated in a similar manner to positional bias, by augmenting the attentional input from Eqs 5 and 6 to include:

\[f_{ji} ={\mathbf{v}^{\top}}\tanh\Big{(}\ldots+{\bf W}^{\text{(am)}}\xi_{1}({ \mathbf{\alpha}_{j-1}};i)\Big{)}\] (7) 

where ... abbreviates the \({\mathbf{e}_{i}}\), \({\mathbf{g}_{j-1}}\) and \(\psi\) components from Eq 6, and \(\xi_{1}({\mathbf{\alpha}_{j-1}})\) provides a fixed dimensional representation of the attention state for the preceding word. It is not immediately obvious how to incorporate the previous attention vector as \({\mathbf{\alpha}}\) is dynamically sized to match the source sentence length, thus using it directly would not generalise over sentences of different lengths. For this reason, we make a simplification by just considering local moves offset by \(\pm k\) positions, that is,

\[\xi_{1}({\mathbf{\alpha}_{j-1}};i)=\left[\alpha_{j-1,i-k},..,\alpha_{j-1,i},.., \alpha_{j-1,i+k}\right]^{\top}\]

with \({\bf W}^{\text{(am)}}\in\mathbb{R}^{A\times(2k+1)}\). Our approach is likely to capture the most important alignments patterns forming the backbone of the alignment HMM, namely monotone, 1-to-many, and local inversions.

### Fertility

Fertility is the propensity for a word to be translated as a consistent number of words in the other language, e.g., _Iseseivusdeklaratsioon_ (Et) translates as 3-4 words in English, namely _(the) Declaration of Independence_. Fertility is a central component in the IBM models 3-5 Brown et al. (1993). Incorporating fertility into the attentional model is a little more involved, and we present two techniques for doing so.

Local fertilityFirst we consider a feature-based technique, which includes the following features

\[\xi_{2}({\mathbf{\alpha}_{<j}};i)=\left[\sum_{j^{\prime}<j}\alpha_{j^{\prime},i-k },..,\sum_{j^{\prime}<j}\alpha_{j^{\prime},i},..,\sum_{j^{\prime}<j}\alpha_{ j^{\prime},i+1}\right]^{\top}\]

and the corresponding feature weights, i.e., \({\bf W}^{\text{(af)}}\in\mathbb{R}^{A\times(2k+1)}\). These sums represent the total alignment score for the surrounding source words, similar to fertility in a traditional latent variable model, which is the sum over binary alignment random variables. A word which already has several alignments can be excluded from participating in more alignments, thus combating the garbage collection problem. Conversely words that tend to need high fertility can be learned through the interactions between these features and the word and context embeddings in Eq 7.

Global fertilityA second, more explicit, technique for incorporating fertility is to include this as a modelling constraint. Initially we considered a soft constraint based on the approach in Xu et al. (2015), where an image captioning model was biased to attend to every pixel in the image exactly once. In our setting, the same idea can be applied through adding a regularisation term to the training objective of the form \(\sum_{i}\Big{(}1-\sum_{j}\alpha_{j,i}\Big{)}^{2}\). However this method is overly restrictive: enforcing that every word is used exactly once is not appropriate in translation where some words are likely to be dropped (e.g., determiners and other function words), while others might need to be translated several times to produce a phrase in the target language.3 For this reason we develop an alternative method, based around a contextual fertility model, \(p(f_{i}|{\mathbf{s}},i)=\mathcal{N}\left(\mu(e_{i}),\sigma^{2}(e_{i})\right)\) which scores the fertility of source word \(i\), defined as \(f_{i}=\sum_{j}\alpha_{j,i}\), using a normal distribution4 parameterised by \(\mu\) and \(\sigma^{2}\), both positive scalar valued non-linear functions of the source word encoding \(e_{i}\). This is incorporated into the training objective as an additional additive term, \(\sum_{i}\log p(f_{i}|{\mathbf{s}},i)\), for each training sentence.

Footnote 3: Modern decoders Koehn et al. (2003) often impose the restriction of each word being translated exactly once, however this is tempered by their use of phrases as translation units rather than words, which allow for higher fertility within phrases.

Footnote 4: The normal distribution is deficient, as it has support for all scalar values, despite \(f_{i}\) being bounded above and below (\(0\leq f_{i}\leq J\)). This could be corrected by using a truncated normal, or various other choices of distribution.

This formulation allows for greater consistency in translation, through e.g., learning which words tend to be omitted from translation, or translate as several words. Compared to the fertility model in IBM 3-5 Brown et al. (1993), ours uses many fewer parameters through working over vector embeddings, and moreover, the BiRNN encoding of the source means that we learn context-dependent fertilities, which can be useful for dealing with fixed syntactic patterns or multi-word expressions.

### Bilingual Symmetry

So far we have considered a conditional model of the target given the source, modelling \(p({\mathbf{t}}|{\mathbf{s}})\). However it is well established for latent variable translation models that the alignments improve if \(p({\mathbf{s}}|{\mathbf{t}})\) is 

also modelled and the inferences of both directional models are combined - evidenced by the symmetrisation heuristics used in most decoders Koehn et al. (2005), and also by explicit joint agreement training objectives Liang et al. (2006); Ganchev et al. (2008). The rationale is that both models make somewhat independent errors, so an ensemble stands to gain from variance reduction.

We propose a method for joint training of two directional models as pictured in Figure 2. Training twinned models involves optimising \({\cal L}=-\log p(\mathbf{t}|\mathbf{s})-\log p(\mathbf{s}|\mathbf{t})+\gamma B\) where, as before, we consider only a single sentence pair, for simplicity of notation. This corresponds to a pseudo-likelihood objective, with the \(B\) linking the two models.5 The \(B\) component considers the alignment (attention) matrices, \(\mathbf{\alpha}^{s\to t}\in\mathbb{R}^{J\times I}\) and \(\mathbf{\alpha}^{t\gets s}\in\mathbb{R}^{I\times J}\), and attempts to make these close to one another for both translation directions (see Fig. 2). To achieve this, we use a 'trace bonus', inspired by Levinboim et al. (2015), formulated as

Footnote 5: We could share some parameters, e.g., the word embedding matrices, however we found this didn’t make much difference versus using disjoint parameter sets. We set \(\gamma=1\) herein.

\[B=-\operatorname{tr}(\mathbf{\alpha}^{s\gets t~{}\top}\mathbf{\alpha}^{s\to t})= \sum_{j}\sum_{i}\alpha_{i,j}^{s\gets t}\alpha_{j,i}^{s\gets t}\,.\]

As the alignment cells are normalised using the \(\mathrm{softmax}\) and thus take values in [0,1], the trace term is bounded above by \(\min(I,J)\) which occurs when the two alignment matrices are transposes of each other, representing perfect one-to-one alignments in both directions

## 4 Experiments

Datasets.We conducted our experiments with four language pairs, translating between English \(\hookrightarrow\) Romanian, Estonian, Russian and Chinese. These languages were chosen to represent a range of translation difficulties, including languages with significant morphological complexity (Estonian, Russian). We focus on a (simulated) low resource setting, where only a limited amount of training data is available. This serves to demonstrate the robustness and generalisation of our model on sparse data - something that has not yet been established for neural models with millions of parameters with vast potential for over-fitting.

Table 1 shows the statistics of the training sets.6 For Chinese-English, the data comes from the BTEC corpus, where the number of training sentence pairs is 44,016. We used 'devset1_2' and 'devset_3' as the development and test sets, respectively, and in both cases used only the first reference for evaluation. For Romanian and Estonian, the data come from the Europarl corpus Koehn (2005), where we used 100K sentence pairs for training, and 3K for development and 2K for testing.7 The Russian-English data was taken from a web derived corpus Antonova and Misyurev (2011). The dataset is split into three parts using the same technique as for the Europarl sets. During the preprocessing stage we lower-cased and tokenized the data, and excluded sentences longer than 30 words. For the Europarl

Figure 2: Symmetric training with trace bonus, computed as matrix multiplication, \(-\operatorname{tr}(\mathbf{\alpha}^{s\leftarrow t}\mathbf{\alpha}^{s\gets t~{} \top})\). Dark shading indicates higher values.

 

data, we also removed sentences containing headings and other meeting formalities.8

Footnote 7: [https://github.com/clab/cnn/](https://github.com/clab/cnn/)

Models and Baselines.We have implemented our neural translation model with linguistic features in C++ using the CNN library.9 We compared our proposed model against our implementations of the attentional model Bahdanau et al. (2015) and encoder-decoder architecture Sutskever et al. (2014). As the baseline, we used a state-of-the-art phrase-based statistical machine translation model built using Moses Koehn et al. (2007) with the standard features: relative-frequency and lexical translation model probabilities in both directions; distortion model; language model and word count. We used KenLM Heafield (2011) to create 3-gram language models with Kneser-Ney smoothing on the target side of the bilingual training corpora.

Footnote 7: E.g., _(The sitting was closed at 10.20pm)_.

Footnote 8: [https://github.com/clab/cnn/](https://github.com/clab/cnn/)

Evaluation Measures.Following previous work Kalchbrenner and Blunsom (2013); Sutskever et al. (2014); Bahdanau et al. (2015); Neubig et al. (2015), we evaluated all neural models using test set perplexities and translation results, as well as in an additional re-ranking setting, using BLEU Papineni et al. (2002) measure. We applied bootstrap resampling Koehn (2004) to measure statistical significance, \(p<0.05\), of our models compared to a baseline. For re-ranking, we generated 100-best translations using the baseline phrase-based model, to which we added log probability features from our neural models alongside all the features of the underlying phrase-based model. We trained the re-ranking models using MERT Och (2003) on development sets with 100-best translations.

### Analysis of Alignment Biases

We start by investigating the effect of various linguistic constraints, described in Section 3, on the attentional model. Table 2 presents the perplexity of trained models for Chinese\(\rightarrow\)English translation. For comparison, we report the results of an encoder-decoder-based neural translation model Sutskever et al. (2014) as the baseline. All other results are for the attentional model with a single-layer LSTM as encoder and two-layer LSTM as decoder, using 512 embedding, 512 hidden, and 256 alignment dimensions. For each model, we also report the number of its parameters. Models are trained end-to-end using stochastic gradient descent (SGD), allowing up to 20 epochs. We use a held-out development set for regularisation by early stopping, which terminated the training after 5-10 epochs for most cases.

As expected, the vanilla attentional model greatly improves over encoder-decoder (perplexity of 4.77 vs. 5.35), clearly making good use of the additional context. Adding the combined positional bias, local fertility, and Markov structure (denoted by +align) further decreases the perplexity to 4.56. Adding the global fertility (+glofer) is detrimental, however, increases perplexity to 5.20. Interestingly, global fertility helps to reduce the perplexity (to 4.31) when used with the pre-training setting (+align+glofer

Figure 3: Perplexity with training epochs on ro-en translation, comparing several model variants.

 

pre). In this case, it is refining an already excellent model from which reliable global fertility estimates can be obtained. This finding is consistent with the other languages, see Figure 3 which shows typical learning curves of different variants of the attentional model. Note that when global fertility is added to the vanilla attentional model with alignment features, it significantly slows down training as it limits exploration in early training iterations, however it does bring a sizeable win when used to fine-tune a pre-trained model. Finally, the bilingual symmetry also helps to reduce the perplexity scores when used with the alignment features, however, does not combine well with global fertility (+align+sym+glofer-pre). This is perhaps an unsurprising result as both methods impose a often-times similar regularising effect over the attention matrix.

Figure 4 illustrates the different attention matrices inferred by the various model variants. Note the difference between the base attentional model and its variant with alignment features ('+align'), where more weight is assigned to diagonal and 1-to-many alignments. Global fertility pushes more attention to the sentinel symbols \(\langle{\rm s}\rangle\) and \(\langle{\rm fs}\rangle\). Determiners and prepositions in English show much lower fertility than nouns, while Estonian nouns have even higher fertility. This accords with Estonian morphology, wherein nouns are inflected with rich case marking, e.g., _noukoguga_ has the cogitative _-ga_ suffix, meaning 'with', and thus translates as several English words (_with the council_). The right-most column corresponds to joint symmetric training, with many more confident attention values especially for consistent 1-to-many alignments (_difficult_ in English and _raskeid_ in Estonian, an adjective in partitive case meaning _some difficult_).

Figure 4: Example development sentence, showing the inferred attention matrix for various models for Et \(\leftrightarrow\) En. Rows correspond to the translation direction and columns correspond to different models: attentional, with alignment features (+align), global fertility (+glofer), and symmetric joint training (+sym). Darker shades denote higher values and white denotes zero.

 



### Experimental Results

The perplexity results of the neural models for the two translation directions across the four language pairs are presented in Table 3. In all cases, our work achieves lower perplexities compared to the vanilla attentional model and the encoder-decoder architecture, owing to the linguistic constraints. We also obtained similar patterns of improvements when decoding, using a greedy decoding strategy, as shown in Table 4. The exception was for en\(\rightarrow\)ru, where the addition of the global fertility (in addition to the other alignment features) was detrimental, resulting in a decrease in BLEU score (5.94\(\rightarrow\)5.26). This may be due to highly noisy nature of the web text corpus of Russian-English language pair, compared to the much cleaner sources for the other language pairs.

Greedy decoding does not appear to be competitive for neural models trained on small parallel corpora, not reaching the level of a phrase-based baseline (see Table 5). Despite this, however, these models still provide substantial gains when used for reranking (as shown in Table 5) for translating into English from the other four languages. We compare re-ranking settings using the log probabilities produced by our model as additional features10 vs. using log probabilities from the vanilla attentional model and the encoder-decoder. The re-rankers based on our model are significantly better than the rest for Chinese and Estonian, and on par with the other for Russian and Romanian\(\rightarrow\)English. In all cases our model has performance at least 1 BLEU point better than the baseline phrase-based system. It is worth noting that for Chinese-English, our re-ranker leads to a substantial increase of almost 3 BLEU points.

Footnote 10: We include two features: the normalised log-probability of the translation, evaluated in both translation directions.

## 5 Related Work

Kalchbrenner and Blunsom (2013) were the first to propose a full neural model of translation, using a convolutional network as the source encoder, followed by an RNN decoder to generate the target translation. This was extended in Sutskever et al. (2014), who replaced the source encoder with an RNN using a Long Short-Term Memory (LSTM) and leveraged the last hidden RNN states as source context for generating the output. Inspired by this, Bahdanau et al. (2015) introduced the notion of "attention" to the model, whereby the source context can dynamically change during the decoding process to attend to the most relevant parts of the source sentence. Further, Luong et al. (2015) refined the attention mechanism to be more local, by constraining attention to a text span, whose words' representations are averaged.

Similar in spirit to our work, recent research has proposed different ways of leveraging the attention history to incorporate alignment structural biases. Luong et al. (2015) made use of the attention vector of the previous position when generating the attention vector for the next position. Feng et al. (2016)

\begin{table}
\begin{tabular}{|l||c c c c|} \hline Lang. Pair & Zh-En & Ru-En & Et-En & Ro-En \\ \hline \hline Enc-Dec & \(17.4\) & \(3.63\) & \(12.5\) & \(21.2\) \\ Attentional & \(29.9\) & \(8.11\) & \(19.7\) & \(33.0\) \\ Our Work & \(\mathbf{31.56^{\bigtriangleup}}\) & \(\mathbf{9.14^{\bigtriangleup}}\) & \(\mathbf{20.44^{\bigtriangleup}}\) & \(\mathbf{34.16^{\bigtriangleup}}\) \\ \hline \hline Lang. Pair & En-Zh & En-Ru & En-Et & En-Ro \\ \hline \hline Enc-Dec & \(14.6\) & \(2.08\) & \(7.97\) & \(16.6\) \\ Attentional & \(20.9\) & \(5.26\) & \(12.5\) & \(28.1\) \\ Our Work & \(\mathbf{23.45^{\bigtriangleup}}\) & \(5.26\) & \(\mathbf{13.40^{\bigtriangleup}}\) & \(\mathbf{30.07^{\bigtriangleup}}\) \\ \hline \end{tabular}
\end{table}
Table 4: BLEU scores on the test sets for the two translation directions, using greedy decoding. **bold:** Best performance, \({}^{\blacklozenge}\): Statistically significantly better than Attenational.

\begin{table}
\begin{tabular}{|l||c c c c|} \hline Lang. Pair & Zh-En & Ru-En & Et-En & Ro-En \\ \hline \hline Enc-Dec & \(5.35\) & \(61.9\) & \(18.2\) & \(10.3\) \\ Attentional & \(4.77\) & \(41.7\) & \(12.8\) & \(6.62\) \\ Our Work & \(\mathbf{4.31}\) & \(\mathbf{39.9}\) & \(\mathbf{11.8}\) & \(\mathbf{5.89}\) \\ \hline \hline Lang. Pair & En-Zh & En-Ru & En-Et & En-Ro \\ \hline \hline Enc-Dec & \(8.60\) & \(67.3\) & \(31.4\) & \(11.5\) \\ Attentional & \(7.49\) & \(43.0\) & \(19.4\) & \(7.30\) \\ Our Work & \(\mathbf{6.24}\) & \(\mathbf{40.6}\) & \(\mathbf{17.0}\) & \(\mathbf{6.35}\) \\ \hline \end{tabular}
\end{table}
Table 3: Perplexity on the test sets for the two translation directions. Our work includes: bidirectional LSTM attentional model combined with positional bias, Markov, local fertility, and global fertility (pre-trained setting).



added another recurrent structure for the attention mechanism to enhance its memorization capabilities and capture long-range dependencies between the attention vectors. Tu et al. (2016) proposed a coverage vector to keep track of the attention history, hence refining future attentions. Finally, Cheng et al. (2015) proposed a similar agreement-based joint training for bidirectional attention-based neural machine translation, and showed significant improvements in BLEU for the large data French\(\leftrightarrow\)English translation.

## 6 Conclusion

We have shown that the attentional model of translation does not capture many well known properties of traditional word-based translation models, and proposed several ways of imposing these as structural biases on the model. We show improvements across several challenging language pairs in a low-resource setting, as well as in perplexity, translation and reranking evaluations. In future work we intend to investigate the model performance on larger-scale datasets, and incorporate further linguistic information, such as morphological representations.

### Acknowledgements

The work reported here was started at JSALT 2015 in UW, Seattle, and was supported by JHU via grants from NSF (IIS), DARPA (LORELEI), Google, Microsoft, Amazon, Mitsubishi Electric, and MERL. Dr Cohn was supported by the ARC (Future Fellowship).

## References

* [Antonova and Misyurev2011] Alexandra Antonova and Alexey Misyurev. 2011. Building a web-based parallel corpus and filtering out machine-translated text. In _Proceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web_, pages 136-144.
* [Bahdanau et al.2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _Proceedings of the International Conference on Learning Representations (ICLR)_, San Diego, CA.
* [Brown et al.1993] Peter E. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. _Computational Linguistics_, 19(2).
* [Cheng et al.2015] Yong Cheng, Shiqi Shen, Zhongjun Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2015. Agreement-based joint training for bidirectional attention-based neural machine translation. In _arXiv: 1512.04650 [cs.CL]_.
* [Cho et al.2014] K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio. 2014. On the properties of neural machine translation. In _arXiv:1409.1259 [cs.CL]_.
* [Dyer et al.2013] Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 644-648, Atlanta, Georgia, June. Association for Computational Linguistics.
* [Feng et al.2016] S. Feng, S. Liu, M. Li, and M. Zhou. 2016. Implicit Distortion and Fertility Models for Attention-based Encoder-Decoder NMT Model. _ArXiv e-prints_, January.
* [Ganchev et al.2008] Kuzman Ganchev, Joao V. Graca, and Ben Taskar. 2008. Better alignments = better translations? In _Proceedings of ACL-08: HLT_, pages 986-993, Columbus, Ohio, June. Association for Computational Linguistics.
* [Heafield2011] Kenneth Heafield. 2011. KenLM: faster and smaller language model queries. In _Proceedings of the EMNLP 2011 Sixth Workshop on Statistical Machine Translation_, pages 187-197, Edinburgh, Scotland, United Kingdom, July.
* [Hochreiter and Schmidhuber1997] S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. _Neural Computation_, 9:1735-1780.
* [Kalchbrenner and Blunsom2013] Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, Seattle, October.
* [Koehn et al.2003] Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In _Proceedings of North American Chapter of the Association for Computational Linguistics on Human Language Technology_, pages 48-54.
* [Koehn et al.2005] Philipp Koehn, Amittai Axelrod, Alexandra Birch, Chris Callison-Burch, Miles Osborne, David Talbot, and Michael White. 2005. Edinburgh system description for the 2005 IWSLT speech translation evaluation. In _IWSLT_, pages 68-75.
* [Koehn et al.2007] Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In _Proc. ACL Interactive Poster and Demonstration sessions_, pages 177-180 

Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Dekang Lin and Dekai Wu, editors, _Proceedings of EMNLP 2004_, pages 388-395, Barcelona, Spain, July. Association for Computational Linguistics.
* [Koehn2005] Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In _Conference Proceedings: the tenth Machine Translation Summit_, pages 79-86. AAMT.
* [Levinboim et al.2015] Tomer Levinboim, Ashish Vaswani, and David Chiang. 2015. Model invertibility regularization: Sequence alignment with or without parallel data. In _Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL)_, pages 609-618, Denver, CO.
* [Liang et al.2006] Percy Liang, Ben Taskar, and Dan Klein. 2006. Alignment by agreement. In _Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL)_, pages 104-111, New York, NY.
* [Luong et al.2015] Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1412-1421, Lisbon, Portugal, September. Association for Computational Linguistics.
* [Neubig et al.2015] Graham Neubig, Makoto Morishita, and Satoshi Nakamura. 2015. Neural reranking improves subjective quality of machine translation: NAIST at WAT2015. In _Proceedings of the 2nd Workshop on Asian Translation (WAT2015)_, Kyoto, Japan, October.
* [Och and Ney2003] Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. _Computational Linguistics_, 29(1):19-51.
* Volume 1_, ACL.
* [Papineni et al.2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_, ACL.
* [Sutskever et al.2014] Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In _Neural Information Processing Systems (NIPS)_, pages 3104-3112, Montreal.
* [Tu et al.2016] Z. Tu, Z. Lu, Y. Liu, X. Liu, and H. Li. 2016. Modeling Coverage for Neural Machine Translation. _ArXiv e-prints_, January.
* [Vogel et al.1996] Stephan Vogel, Hermann Ney, and Christoph Tillmann. 1996. HMM-based word alignment in statistical translation. In _Proceedings of the International Conference on Computational Linguistics (COLING)_, pages 836-841.
* [Xu et al.2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In _Proceedings of the 32nd International Conference on Machine Learning (ICML-15)_, pages 2048-2057.