Venue: NAACL 2018

# Improving Lexical Choice in Neural Machine Translation

Toan Q. Nguyen   David Chiang

Department of Computer Science and Engineeering

University of Notre Dame

{tnguye28,dchiang}@nd.edu

###### Abstract

We explore two solutions to the problem of mistranslating rare words in neural machine translation. First, we argue that the standard output layer, which computes the inner product of a vector representing the context with all possible output word embeddings, rewards frequent words disproportionately, and we propose to fix the norms of both vectors to a constant value. Second, we integrate a simple lexical module which is jointly trained with the rest of the model. We evaluate our approaches on eight language pairs with data sizes ranging from 100k to 8M words, and achieve improvements of up to +4.3 BLEU, surpassing phrase-based translation in nearly all settings.1

Footnote 1: The code for this work can be found at [https://github.com/tng177/improving_lexical_choice_in_mmt](https://github.com/tng177/improving_lexical_choice_in_mmt)

## 1 Introduction

Neural network approaches to machine translation Sutskever et al. (2014); Bahdanau et al. (2015); Luong et al. (2015a); Gehring et al. (2017) are appealing for their single-model, end-to-end training process, and have demonstrated competitive performance compared to earlier statistical approaches Koehn et al. (2007); Junczys-Dowmunt et al. (2016). However, there are still many open problems in NMT Koehn and Knowles (2017). One particular issue is mistranslation of rare words. For example, consider the Uzbek sentence:

_Source:_ Ammo muammolar hali ko'p, deydi amerikalik olim Entoni Fauchi.

_Reference:_ But still there are many problems, says American scientist Anthony Fauci.

_Baseline NMT:_ But there is still a lot of problems, says James Chan.

At the position where the output should be _Fauci_, the NMT model's top three candidates are _Chan_, _Fauci_, and _Jenner_. All three surnames occur in the training data with reference to immunologists: Fauci is the director of the National Institute of Alergy and Infectious Diseases, Margaret (not James) Chan is the former director of the World Health Organization, and Edward Jenner invented smallbox vaccine. But _Chan_ is more frequent in the training data than _Fauci_, and _James_ is more frequent than either _Anthony_ or _Margaret_.

Because NMT learns word representations in continuous space, it tends to translate words that "seem natural in the context, but do not reflect the content of the source sentence" Arthur et al. (2016). This coincides with other observations that NMT's translations are often fluent but lack accuracy Wang et al. (2017b); Wu et al. (2016).

Why does this happen? At each time step, the model's distribution over output words \(e\) is

\[p(e)\propto\exp\left(W_{e}\cdot\tilde{h}+b_{e}\right)\]

where \(W_{e}\) and \(b_{e}\) are a vector and a scalar depending only on \(e\), and \(\tilde{h}\) is a vector depending only on the source sentence and previous output words. We propose two modifications to this layer. First, we argue that the term \(W_{e}\cdot\tilde{h}\), which measures how well \(e\) fits into the context \(\tilde{h}\), favors common words disproportionately, and show that it helps to fix the norm of both vectors to a constant. Second, we add a new term representing a more direct connection from the source sentence, which allows the model to better memorize translations of rare words.

Below, we describe our models in more detail. Then we evaluate our approaches on eight language pairs, with training data sizes ranging from 100k words to 8M words, and show improvements of up to +4.3 BLEU, surpassing phrase-based translation in nearly all settings. Finally, we provide some analysis to better understand why our modifications work well.





## 2 Neural Machine Translation

Given a source sequence \(f=f_{1}f_{2}\cdots f_{m}\), the goal of NMT is to find the target sequence \(e=e_{1}e_{2}\cdots e_{n}\) that maximizes the objective function:

\[\log p(e\mid f)=\sum_{t=1}^{n}\log p(e_{t}\mid e_{<t},f).\]

We use the _global attentional_ model with _general scoring function_ and _input feeding_ by Luong et al. (2015a). We provide only a very brief overview of this model here. It has an encoder, an attention, and a decoder. The encoder converts the words of the source sentence into _word embeddings_, then into a sequence of _hidden states_. The decoder generates the target sentence word by word with the help of the attention. At each time step \(t\), the attention calculates a set of _attention weights_\(a_{t}(s)\). These attention weights are used to form a weighted average of the encoder hidden states to form a _context vector_\(c_{t}\). From \(c_{t}\) and the hidden state of the decoder are computed the _attentional hidden state_\(\tilde{h}_{t}\). Finally, the predicted probability distribution of the \(t\)'th target word is:

\[p(e_{t}\mid e_{<t},f)=\text{softmax}(W^{o}\tilde{h}_{t}+b^{o}). \tag{1}\]

The rows of the output layer's weight matrix \(W^{o}\) can be thought of as embeddings of the output vocabulary, and sometimes are in fact tied to the embeddings in the input layer, reducing model size while often achieving similar performance (Inan et al., 2017; Press and Wolf, 2017). We verified this claim on some language pairs and found out that this approach usually performs better than without tying, as seen in Table 1. For this reason, we always tie the target embeddings and \(W^{o}\) in all of our models.

## 3 Normalization

The output word distribution (1) can be written as:

\[p(e)\propto\exp\left(||W_{e}||\bar{h}||\cos\theta_{W_{e},\tilde{h}}+b_{e} \right),\]

where \(W_{e}\) is the embedding of \(e\), \(b_{e}\) is the \(e\)'th component of the bias \(b^{o}\), and \(\theta_{W_{e},\tilde{h}}\) is the angle between \(W_{e}\) and \(\tilde{h}\). We can intuitively interpret the terms as follows. The term \(||\bar{h}||\) has the effect of sharpening or flattening the distribution, reflecting whether the model is more or less certain in a particular context. The cosine similarity \(\cos\theta_{W_{e},\tilde{h}}\) measures how well \(e\) fits into the context. The bias \(b_{e}\) controls how much the word \(e\) is generated; it is analogous to the language model in a log-linear translation model Och and Ney (2002).

Finally, \(||W_{e}||\) also controls how much \(e\) is generated. Figure 1 shows that it generally correlates with frequency. But because it is multiplied by \(\cos\theta_{W_{e},\tilde{h}}\), it has a stronger effect on words whose embeddings have direction similar to \(\tilde{h}\), and less effect or even a negative effect on words in other directions. We hypothesize that the result is that the model learns \(||W_{e}||\) that are disproportionately large.

For example, returning to the example from Section 1, these terms are:

\begin{tabular}{l c c c c c}   \[e\] & \[||W_{e}||\] & \[||\tilde{h}||\] & \[\cos\theta_{W_{e},\tilde{h}}\] & \[b_{e}\] & logit \\ \hline Chan & 5.25 & 19.5 & 0.144 & \[-1.53\] & 13.2 \\ Fauci & 4.69 & 19.5 & 0.154 & \[-1.35\] & 12.8 \\ Jenner & 5.23 & 19.5 & 0.120 & \[-1.59\] & 10.7 \\ \end{tabular} Observe that \(\cos\theta_{W_{e},\tilde{h}}\) and even \(b_{e}\) both favor the correct output word _Fauci_, whereas \(||W_{e}||\) favors the more frequent, but incorrect, word _Chan_. The most frequently-mentioned immunologist trumps other immunologists.

To solve this issue, we propose to fix the norm of all target word embeddings to some value \(r\). Following the weight normalization approach of Salimans and Kingma (2016), we reparameterize \(W_{e}\) as \(r\frac{\nu_{e}}{||w_{e}||}\), but keep \(r\) fixed.

A similar argument could be made for \(||\tilde{h}_{t}||\): because a large \(||\tilde{h}_{t}||\) sharpens the distribution, causing frequent words to more strongly dominate rare words, we might want to limit it as well. We compared both approaches on a development set and found that replacing \(\tilde{h}_{t}\) in equation (1) with \(r\frac{\tilde{h}_{t}}{||w_{e}||}\) indeed performs better, as shown in Table 1.

\begin{table}
\begin{tabular}{l l l l} \hline \hline  & ha-en & tu-en & hu-en \\ \hline untied embeddings & 17.2 & 11.5 & 26.5 \\ tied embeddings & 17.4 & 13.8 & 26.5 \\ \hline don’t normalize \(\tilde{h}_{t}\) & 18.6 & 14.2 & 27.1 \\ normalize \(\tilde{h}_{t}\) & 20.5 & 16.1 & 28.8 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Preliminary experiments show that tying target embeddings with output layer weights performs as well as or better than the baseline, and that normalizing \(\tilde{h}\) is better than not normalizing \(\tilde{h}\). All numbers are BLEU scores on development sets, scored against tokenized references.





## 4 Lexical Translation

The attentional hidden state \(\tilde{h}\) contains information not only about the source word(s) corresponding to the current target word, but also the contexts of those source words and the preceding context of the target word. This could make the model prone to generate a target word that fits the context but doesn't necessarily correspond to the source word(s). Count-based statistical models, by contrast, don't have this problem, because they simply don't model any of this context. Arthur et al. (2016) try to alleviate this issue by integrating a count-based lexicon into an NMT system. However, this lexicon must be trained separately using GIZA++ Och and Ney (2003), and its parameters form a large, sparse array, which can be difficult to store in GPU memory.

We propose instead to use a simple feedforward neural network (FFNN) that is trained jointly with the rest of the NMT model to generate a target word based directly on the source word(s). Let \(f_{s}\) (\(s=1,\ldots,m\)) be the embeddings of the source words. We use the attention weights to form a weighted average of the embeddings (not the hidden states, as in the main model) to give an average source-word embedding at each decoding time step \(t\):

\[f^{\ell}_{t}=\tanh\sum_{s}a_{t}(s)f_{s}.\]

Then we use a one-hidden-layer FFNN with skip connections He et al. (2016):

\[h^{\ell}_{t}=\tanh(Wf^{\ell}_{t})+f^{\ell}_{t}\]

and combine its output with the decoder output to get the predictive distribution over output words at time step \(t\):

\[p(y_{t}\mid y_{<t},x)=\text{softmax}(W^{o}\tilde{h}_{t}+b^{o}+W^{\ell}h^{ \ell}_{t}+b^{\ell}).\]

For the same reasons that were given in Section 3 for normalizing \(\tilde{h}_{t}\) and the rows of \(W^{o}_{t}\), we normalize \(h^{\ell}_{t}\) and the rows of \(W^{\ell}\) as well. Note, however, that we do not tie the rows of \(W^{\ell}\) with the word embeddings; in preliminary experiments, we found this to yield worse results.

## 5 Experiments

We conducted experiments testing our normalization approach and our lexical model on eight language pairs using training data sets of various sizes. This section describes the systems tested and our results.

### Data

We evaluated our approaches on various language pairs and datasets:

Figure 1: The word embedding norm \(\|W_{e}\|\) generally correlates with the frequency of \(e\), except for the most frequent words. The bias \(b_{e}\) has the opposite behavior. The plots show the median and range of bins of size 256.



* Tamil (ta), Urdu (ur), Hausa (ha), Turkish (tu), and Hungarian (hu) to English (en), using data from the LORELEI program.
* English to Vietnamese (vi), using data from the IWSLT 2015 shared task.2 Footnote 2: [https://nlp.stanford.edu/projects/nmt/](https://nlp.stanford.edu/projects/nmt/)
* To compare our approach with that of Arthur et al. (2016), we also ran on their English to Japanese (ja) KFTT and BTEC datasets.3

Footnote 3: [http://isw3.naist.jp/-philip-a/emlp2016/](http://isw3.naist.jp/-philip-a/emlp2016/)

We tokenized the LORELEI datasets using the default Moses tokenizer, except for Urdu-English, where the Urdu side happened to be tokenized using Morfessor FlatCat (\(w=0.5\)). We used the preprocessed English-Vietnamese and English-Japanese datasets as distributed by Luong et al., and Arthur et al., respectively. Statistics about our data sets are shown in Table 2.

### Systems

We compared our approaches against two baseline NMT systems:

**untied**, which does not tie the rows of \(W_{o}\) to the target word embeddings, and

**tied**, which does.

In addition, we compared against two other base-line systems:

**Moses**: The Moses phrase-based translation system [Koehn et al., 2007], trained on the same data as the NMT systems, with the same maximum sentence length of 50. No additional data was used for training the language model. Unlike the NMT systems, Moses used the full vocabulary from the training data; unknown words were copied to the target sentence.

**Arthur:** Our reimplementation of the discrete lexicon approach of arthur et al. (2016). We only tried their auto lexicon, using GIZA++ [Och and Ney, 2003], integrated using their bias approach. Note that we also tied embedding as we found it also helped in this case.

Against these baselines, we compared our new systems:

**fixnorm**: The normalization approach described in Section 3.

**fixnorm+lex**: The same, with the addition of the lexical translation module from Section 4.

### Details

ModelFor all NMT systems, we fed the source sentences to the encoder in reverse order during both training and testing, following Luong et al. (2015). Information about the number and size of hidden layers is shown in Table 2. The word embedding size is always equal to the hidden layer size.

Following common practice, we only trained on sentences of 50 tokens or less. We limited the vocabulary to word types that appear no less than 5 times in the training data and map the rest to UNK. For the English-Japanese and English-Vietnamese datasets, we used the vocabulary sizes reported in their respective papers [Arthur et al., 2016; Luong and Manning, 2015].

For **fixnorm**, we tried \(r\in\{3,5,7\}\) and selected the best value based on the development set performance, which was \(r=5\) except for English-Japanese (BTEC), where \(r=7\). For **fixnorm+lex**, because \(W_{s}\bar{h}_{t}+W^{\ell}h_{t}^{\ell}\) takes on values in \([-2r^{2},2r^{2}]\), we reduced our candidate \(r\) values by roughly a factor of \(\sqrt{2}\), to \(r\in\{2,3.5,5\}\). A radius \(r=3.5\) seemed to work the best for all language pairs.

TrainingWe trained all NMT systems with Adadelta [Zeiler, 2012]. All parameters were initialized uniformly from \([-0.01,0.01]\). When a gradient's norm exceeded 5, we normalized it to 5. We also used dropout on non-recurrent connections only [Zaremba et al., 2014], with probability 0.2. We used minibatches of size 32. We trained for 50 epochs, validating on the development set after every epoch, except on English-Japanese, where we validated twice per epoch. We kept the best checkpoint according to its BLEU on the development set.

InferenceWe used beam search with a beam size of 12 for translating both the development and test sets. Since NMT often favors short translations [Cho et al., 2014], we followed Wu et al. (2016) in using a modified score \(s(e\mid f)\) in place of log-probability:

\[s(e\mid f) =\frac{\log p(e\mid f)}{ lp(e)}\] \[lp(e) =\frac{(5+|e|)^{\alpha}}{(5+1)^{\alpha}}\]

We set \(\alpha=0.8\) for all of our experiments.

Finally, we applied a postprocessing step to replace each UNK in the target translation with the 

source word with the highest attention score (Luong et al., 2015).

EvaluationFor translation into English, we report case-sensitive NIST BLEU against detokenized references. For English-Japanese and English-Vietnamese, we report tokenized, case-sensitive BLEU following Arthur et al. (2016) and Luong and Manning (2015). We measure statistical significance using bootstrap resampling (Koehn, 2004).

## 6 Results and Analysis

### Overall

Our results are shown in Table 3. First, we observe, as has often been noted in the literature, that NMT tends to perform poorer than PBMT on low resource settings (note that the rows of this table are sorted by training data size).

Our **fixnorm** system alone shows large improvements (shown in parentheses) relative to **tied**. Integrating the lexical module **(fixnorm+lex)** adds in further gains. Our **fixnorm+lex** models surpass Moses on all tasks except Urdu- and Hausa-English, where it is 1.6 and 0.7 BLEU short respectively.

The method of Arthur et al. (2016) does improve over the baseline NMT on most language pairs, but not by as much and as consistently as our models, and often not as well as Moses. Unfortunately, we could not replicate their approach for English-Japanese (KFTT) because the lexical table was too large to fit into the computational graph.

For English-Japanese (BTEC), we note that, due to the small size of the test set, all systems except for Moses are in fact not significantly different from **tied** (\(p>0.01\)). On all other tasks, however, our systems significantly improve over **tied** (\(p<0.01\)).

### Impact on translation

In Table 4, we show examples of typical translation mistakes made by the baseline NMT systems. In the Uzbek example (top), **untied** and **tied** have confused _34_ with _UNK_ and _700_, while in the Turkish one (middle), they incorrectly output other proper names, _Afghan_ and _Myanmar_, for the proper name _Kenya_. Our systems, on the other hand, translate these words correctly.

The bottom example is the one introduced in Section 1. We can see that our **fixnorm** approach does not completely solve the mistranslation issue, since it translates _Entoni Fauchi_ to _UNK_ (which is arguably better than _James Chan_). On the other hand, **fixnorm+lex** gets this right. To better understand how the lexical module helps in this case, we look at the top five translations for the word _Fauci_ in **fixnorm+lex**:

\begin{tabular}{l c r r r}   \[e\] & \[\cos\theta_{W_{e},h}\] & \[\cos\theta_{W^{t}_{e},h_{l}}\] & \[b_{e}+b^{t}_{e}\] & logit \\ \hline Fauci & \[0.522\] & \[0.762\] & \[-8.71\] & \[7.0\] \\ UKK & \[0.566\] & \[-0.009\] & \[-1.25\] & \[5.6\] \\ Anthony & \[0.263\] & \[0.644\] & \[-8.70\] & \[2.4\] \\ Ahmedova & \[0.555\] & \[0.173\] & \[-8.66\] & \[0.3\] \\ Chan & \[0.546\] & \[0.150\] & \[-8.73\] & \[-0.2\] \\ \end{tabular} As we can see, while \(\cos\theta_{W_{e},h}\) might still be confused between similar words, \(\cos\theta_{W^{t}_{e},h_{l}}\) significantly favors _Fauci_.

### Alignment and unknown words

Both our baseline NMT and **fixnorm** models suffer from the problem of shifted alignments noted by Koehn and Knowles (2017). As seen in Figure 1(a) and 1(b), the alignments for those two systems seem to shift by one word to the left (on the source side). For example, _noi_ should be aligned to _said_ instead of _Telekom_, and so on. Although this is not a problem _per se_, since the decoder can decide to attend to any position in the encoder states as long as the state at that position holds the information the decoder needs, this becomes a real issue when we need to make use of the alignment information, as in unknown word replacement (Luo et al., 2015). As we can see in Figure 2, because of the alignment shift, both **tied** and **fixnorm** incorrectly replace the two unknown words (in bold) with _But Deutsche_ instead of _Deutsche Telekom_. In contrast, under **fixnorm+lex** and the model of Arthur et al. (2016), the alignment is corrected, causing the UNKs to be replaced with the correct source words.

### Impact of \(r\)

The single most important hyper-parameter in our models is \(r\). Informally speaking, \(r\) controls how much surface area we have on the hypersphere to allocate to word embeddings. To better understand its impact, we look at the training perplexity and dev BLEUs during training with different values of \(r\). Table 6 shows the train perplexity and best tokenized dev BLEU on Turkish-English for **fixnorm** and **fixnorm+lex** with different values of \(r\). As we can see, a smaller \(r\) results in 



\begin{table}
\begin{tabular}{l l p{284.5pt}} \hline \hline input & Dushanba kuni Hindistonda karinda**34**kishi halok bo’llgani xabar qilindi . \\ reference & At least **34** more deaths were reported Monday in India . \\
**untied** & At least **UNK** people have died in India on Monday . \\
**tied** & It was reported that at least **700** people died in Monday . \\
**fixnorm+lex** & At least **34** people died in India on Monday . \\ \hline input & Yarin Kenya*da by yardim konferanzaeleneck . \\ reference & Tomorrow a conference for aid will be conducted in **Kenya** . \\
**untied** & Tomorrow there will be an **Afghani relief** conference . \\
**tied** & Tomorrow there will be a relief conference in **Myanmar** . \\
**fixnorm+lex** & Tomorrow it will be a aid conference in **Kenya** . \\ \hline input & Ammo muamolar hali Ko’p , deydi amerklakil olim **Entoni Fauchi** . \\ reference & But still there are many problems , says American scientist **Anthony Fauci** . \\
**untied** & But there is still a lot of problems , says **James Chan** . \\
**tierd** & However , there is still a lot of problems , says American scientists . \\
**fixnorm** & But there is still a lot of problems , says American scientist **UNK UNK** . \\
**fixnorm+lex** & But there are still problems , says American scientist **Annthony Fauci** . \\ \hline \hline \end{tabular}
\end{table}
Table 4: Example translations, in which **untied** and **tied** generate incorrect, but often semantically related, words, but **fixnorm** and/or **fixnorm+lex** generate the correct ones.

\begin{table}
\begin{tabular}{l r c c c c c} \hline \hline  & **untied** & **tied** & **fixnorm** & **fixnorm+lex** & **Moses** & **Arthur** \\ \hline ta-en & 10.3 & 11.1 & 14 (+2.9) & 15.3 & (+4.2) & 10.5 (\(-\)0.6) & 14.1 (+3.0) \\ ur-en & 7.9 & 10.7 & 12 (+1.3) & 13 & (+2.3) & 14.6 (+3.9) & 12.5 (+1.8) \\ ha-en & 16.0 & 16.6 & 20 (+3.4) & 21.5 & (+4.9) & 22.2 (+5.6) & 18.7 (+2.1) \\ tu-en & 12.2 & 12.6 & 16.4 (+3.8) & 19.1 & (+6.5) & 18.1 (+5.5) & 16.3 (+3.7) \\ uz-en & 14.9 & 15.7 & 18.2 (+2.5) & 19.3 & (+3.6) & 17.2 (+1.5) & 17.1 (+1.4) \\ hu-en & 21.6 & 23.0 & 24.0 (+1.0) & 25.3 & (+2.3) & 21.3 (\(-\)1.7) & 22.7 (-0.3)† \\ en-vi & 25.1 & 25.3 & 26.8 (+1.5) & 27 & (+1.7) & 26.7 (+1.4) & 26.2 (+0.9) \\ en-ja (BTEC) & 51.2 & 53.7 & 52.9 (-0.8)† [FOOTNOTE:†]Footnote †:


* [14] M. A. B. K., and J. S. R. C. Smith. The effect of the effects of the model on the performance of a model in an urban environment. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 207-218, 2016.
* [15] D. L. Bertsekas, P. F. G. T. Chan, and H. E. Stanley. An empirical study of the impact of the model's influence on the 

worse training perplexity, indicating underfitting, whereas if \(r\) is too large, the model achieves better training perplexity but decrased dev BLEU, indicating overfitting.

### Lexicon

One byproduct of **lex** is the lexicon, which we can extract and examine simply by feeding each source word embedding to the FFNN module and calculating \(p^{t}(y)=\text{softmax}(W^{t}h^{t}+b_{\ell})\). In Table 5, we show the top translations for some entries in the lexicons extracted from **fixnorm+lex** for Hungarian, Turkish, and Hausa-English. As expected, the lexical distribution is sparse, with a few top translations accounting for the most probability mass.

### Byte Pair Encoding

BytePair-Encoding (BPE) [10] is commonly used in NMT to break words into word-pieces, improving the translation of rare words. For this reason, we reran our experiments using BPE on the LORELEI and English-Vietnamese datasets. Additionally, to see if our proposed methods work in high-resource scenarios, we run on the WMT 2014 English-German (en-de) dataset,4 using _newstest2013_ as the development set and reporting tokenized, case-sensitive BLEU on _newstest2014_ and _newstest2015_.

Footnote 4: [https://nlp.stanford.edu/projects/nmt/](https://nlp.stanford.edu/projects/nmt/)

We validate across different numbers of BPE operations; specifically, we try {1k, 2k, 3k} merge operations for ta-en and ur-en due to their small sizes, {10k, 12k, 15k} for the other LORELEI datasets and en-vi, and 32k for en-de. Using BPE results in much smaller vocabulary sizes, so we do not apply a vocabulary cut-off. Instead, we train on an additional copy of the training data in which all types that appear once are replaced with UNK, and halve the number of epochs accordingly. Our models, training, and evaluation processes are largely the same, except that for en-de, we use a 4-layer decoder and 4-layer bidirectional encoder (2 layers for each direction).

Table 7 shows that our proposed methods also significantly improve the translation when used with BPE, for both high and low resource language pairs. With BPE, we are only behind Moses on Undu-English.

## 7 Related Work

The closest work to our **lex** model is that of Arthur et al. (2016), which we have discussed already in Section 4. Recent work by Liu et al. (2016) has very similar motivation to that of our **fixnorm** model. They reformulate the output layer in terms of directions and magnitudes, as we do here. Whereas we have focused on the magnitudes, they focus on the directions, modifying the loss function to try to learn a classifier that separates the classes' directions with something like a margin. Wang et al. (2017) also make the same observation that we do for the **fixnorm** model, but for the task of face verification.

Handling rare words is an important problem for NMT that has been approached in various ways. Some have focused on reducing the number of UNKs by enabling NMT to learn from a larger vocabulary [8, 9, 10]; others have focused on replacing UNKs by copying source words [11, 12, 13]. However, these methods only help with unknown words, not rare words. An approach that addresses both unknown and rare words is to use subword-level information [10, 12, 13]. Our approach is different in that we try to identify and address the root of the rare word problem. We expect that our models would benefit from more advanced UNK replacement or subword-level techniques as well.

Recently, Liu and Kirchhoff (2018) have shown that their baseline NMT system with BPE already outperforms Moses for low-resource translation. However, in their work, they use the Transformer network [10], which is quite different from our baseline model. It would be interesting to see if our methods benefit the Trans

\begin{table}
\begin{tabular}{l c r r} \hline \hline system & \(r\) & train ppl & dev BLEU \\ \hline  & 3 & 3.9 & 13.6 \\
**fixnorm** & 5 & 2.5 & 16.1 \\  & 7 & 2.3 & 14.4 \\ \hline  & 2 & 4.2 & 12.3 \\
**fixnorm+lex** & 3.5 & 2.0 & 17.5 \\  & 5 & 1.4 & 16.0 \\ \hline \hline \end{tabular}
\end{table}
Table 6: When \(r\) is too small, high train perplexity and low dev BLEU indicate underfitting; when \(r\) is too large, low train perplexity and low dev BLEU indicate overfitting.

 

former network and other models as well.

## 8 Conclusion

In this paper, we have presented two simple yet effective changes to the output layer of a NMT model. Both of these changes improve translation quality substantially on low-resource language pairs. In many of the language pairs we tested, the baseline NMT system performs poorly relative to phrase-based translation, but our system surpasses it (when both are trained on the same data). We conclude that NMT, equipped with the methods demonstrated here, is a more viable choice for low-resource translation than before, and are optimistic that NMT's repertoire will continue to grow.

## Acknowledgements

This research was supported in part by University of Southern California subcontract 67108176 under DARPA contract HR0011-15-C-0115. Nguyen was supported in part by a fellowship from the Vietnam Education Foundation. We would like to express our great appreciation to Sharon Hu for letting us use her group's GPU cluster (supported by NSF award 1629914), and to NVIDIA corporation for the donation of a Titan X GPU. We also thank Tomer Levinboim for insightful discussions.

## References

* Arthur et al. (2016) Philip Arthur, Graham Neubig, and Satoshi Nakamura. 2016. Incorporating discrete translation lexicons into neural machine translation. In _Proc. EMNLP_.
* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _Proc. ICLR_.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In _Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation_.
* Chung et al. (2016) Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A character-level decoder without explicit segmentation for neural machine translation. In _Proc. ACL_.
* Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. arXiv:1705.03122.
* Gu et al. (2016) Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In _Proc. ACL_.
* Gulcehre et al. (2016) Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. In _Proc. ACL_.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _Proc. CVPR_.
* Inan et al. (2017) Hakan Inan, Khashayar Khosravi, and Richard Socher. 2017. Tying word vectors and word classifiers: A loss framework for language modeling. In _Proc. ICLR_.
* Jean et al. (2015) Sebastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In _Proc. ACL-IJCNLP_.
* Junczys-Dowmunt et al. (2016) Marcin Junczys-Dowmunt, Tomasz Drojak, and Hieu Hoang. 2016. Is neural machine translation ready for deployment? A case study on 30 translation directions. In _Proc. IWSLT_.
* Koehn (2004) Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In _Proc. EMNLP_.
* Koehn et al. (2016)

\begin{table}
\begin{tabular}{l r r r r} \hline \hline  & ** tied** & **fixnorm** & **fixnorm+lex** \\ \hline ta-en & 13 & 15 (+2.0) & 15.9 & (+2.9) \\ ur-en & 10.5 & 12.3 (+1.8) & 13.7 & (+3.2) \\ ha-en & 18 & 21.7 (+3.7) & 22.3 & (+4.3) \\ tu-en & 19.3 & 21 (+1.7) & 22.2 & (+2.9) \\ uz-en & 18.9 & 19.8 (+0.9) & 21 & (+2.1) \\ hu-en & 25.8 & 27.2 (+1.4) & 27.9 & (+2.1) \\ en-vi & 26.3 & 27.3 (+1.0) & 27.5 & (+1.2) \\ en-de (newstest2014) & 19.7 & 22.2 (+2.5) & 20.4 & (+0.7) \\ en-de (newstest2015) & 22.5 & 25 (+2.5) & 23.2 & (+0.7) \\ \hline \hline \end{tabular}
\end{table}
Table 7: Test BLEU for all BPE-based systems. Our models significantly improve over the baseline (\(p<0.01\)) for both high and low resource when using BPE.

 

Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In _Proc. ACL_.
* Koehn and Knowles (2017) Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In _Proc. Workshop on Neural Machine Translation_.
* Liu and Kirchhoff (2018) Angli Liu and Katrin Kirchhoff. 2018. Context models for oov word translation in low-resource languages. In _Proceedings of AMTA 2018, vol. 1: MT Research Track_. AMTA.
* Liu et al. (2016) Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. 2016. Large-margin softmax loss for convolutional neural networks. In _Proc. ICML_.
* Luong and Manning (2015) Minh-Thang Luong and Christopher D. Manning. 2015. Stanford neural machine translation systems for spoken language domain. In _Proc. IWSLT_.
* Luong and Manning (2016) Minh-Thang Luong and Christopher D. Manning. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. In _Proc. ACL_.
* Luong et al. (2015a) Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015a. Effective approaches to attention-based neural machine translation. In _Proc. EMNLP_.
* Luong et al. (2015b) Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba. 2015b. Addressing the rare word problem in neural machine translation. In _Proc. ACL-IJCNLP_.
* Mi et al. (2016) Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. 2016. Vocabulary manipulation for neural machine translation. In _Proc. ACL_.
* Och and Ney (2002) Franz Josef Och and Hermann Ney. 2002. Discriminative training and maximum entropy models for statistical machine translation. In _Proc. ACL_.
* Och and Ney (2003) Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. _Computational Linguistics_ 29(1).
* Press and Wolf (2017) Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In _Proc. EACL_.
* Salimans and Kingma (2016) T. Salimans and D. P. Kingma. 2016. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. _ArXiv e-prints_ .
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _Proc. ACL_.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _NIPS 27_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_. pages 6000-6010.
* Wang et al. (2017a) Feng Wang, Xiang Xiang, Jian Cheng, and Alan L. Yuille. 2017a. Normface: L2 hypersphere embedding for face verification. In _Proceedings of the 25th ACM international conference on Multimedia_. ACM. [https://doi.org/https://doi.org/10.1145/3123266.3123359](https://doi.org/https://doi.org/10.1145/3123266.3123359).
* Wang et al. (2017b) Xing Wang, Zhengdong Lu, Zhaopeng Tu, Hang Li, Deyi Xiong, and Min Zhang. 2017b. Neural machine translation advised by statistical machine translation. In _Proc. AAAI_.
* Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144.
* Zaremba et al. (2014) Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. arXiv:1409.2329.
* Zeiler (2012) Matthew D. Zeiler. 2012. ADADELTA: An adaptive learning rate method. arXiv:1212.5701v1.