Introduction
Neural machine translation (NMT, § SECREF2 ; kalchbrenner13emnlp, sutskever14nips) is a variant of statistical machine translation (SMT; brown93cl), using neural networks. NMT has recently gained popularity due to its ability to model the translation process end-to-end using a single probabilistic model, and for its state-of-the-art performance on several language pairs BIBREF0 , BIBREF1 .
One feature of NMT systems is that they treat each word in the vocabulary as a vector of continuous-valued numbers. This is in contrast to more traditional SMT methods such as phrase-based machine translation (PBMT; koehn03phrasebased), which represent translations as discrete pairs of word strings in the source and target languages. The use of continuous representations is a major advantage, allowing NMT to share statistical power between similar words (e.g. “dog” and “cat”) or contexts (e.g. “this is” and “that is”). However, this property also has a drawback in that NMT systems often mistranslate into words that seem natural in the context, but do not reflect the content of the source sentence. For example, Figure FIGREF2 is a sentence from our data where the NMT system mistakenly translated “Tunisia” into the word for “Norway.” This variety of error is particularly serious because the content words that are often mistranslated by NMT are also the words that play a key role in determining the whole meaning of the sentence.
In contrast, PBMT and other traditional SMT methods tend to rarely make this kind of mistake. This is because they base their translations on discrete phrase mappings, which ensure that source words will be translated into a target word that has been observed as a translation at least once in the training data. In addition, because the discrete mappings are memorized explicitly, they can be learned efficiently from as little as a single instance (barring errors in word alignments). Thus we hypothesize that if we can incorporate a similar variety of information into NMT, this has the potential to alleviate problems with the previously mentioned fatal errors on low-frequency words.
In this paper, we propose a simple, yet effective method to incorporate discrete, probabilistic lexicons as an additional information source in NMT (§ SECREF3 ). First we demonstrate how to transform lexical translation probabilities (§ SECREF7 ) into a predictive probability for the next word by utilizing attention vectors from attentional NMT models BIBREF2 . We then describe methods to incorporate this probability into NMT, either through linear interpolation with the NMT probabilities (§ UID10 ) or as the bias to the NMT predictive distribution (§ UID9 ). We construct these lexicon probabilities by using traditional word alignment methods on the training data (§ SECREF11 ), other external parallel data resources such as a handmade dictionary (§ SECREF13 ), or using a hybrid between the two (§ SECREF14 ).
We perform experiments (§ SECREF5 ) on two English-Japanese translation corpora to evaluate the method's utility in improving translation accuracy and reducing the time required for training.
Neural Machine Translation
The goal of machine translation is to translate a sequence of source words INLINEFORM0 into a sequence of target words INLINEFORM1 . These words belong to the source vocabulary INLINEFORM2 , and the target vocabulary INLINEFORM3 respectively. NMT performs this translation by calculating the conditional probability INLINEFORM4 of the INLINEFORM5 th target word INLINEFORM6 based on the source INLINEFORM7 and the preceding target words INLINEFORM8 . This is done by encoding the context INLINEFORM9 a fixed-width vector INLINEFORM10 , and calculating the probability as follows: DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are respectively weight matrix and bias vector parameters.
The exact variety of the NMT model depends on how we calculate INLINEFORM0 used as input. While there are many methods to perform this modeling, we opt to use attentional models BIBREF2 , which focus on particular words in the source sentence when calculating the probability of INLINEFORM1 . These models represent the current state of the art in NMT, and are also convenient for use in our proposed method. Specifically, we use the method of luong15emnlp, which we describe briefly here and refer readers to the original paper for details.
First, an encoder converts the source sentence INLINEFORM0 into a matrix INLINEFORM1 where each column represents a single word in the input sentence as a continuous vector. This representation is generated using a bidirectional encoder INLINEFORM2
Here the INLINEFORM0 function maps the words into a representation BIBREF3 , and INLINEFORM1 is a stacking long short term memory (LSTM) neural network BIBREF4 , BIBREF5 , BIBREF6 . Finally we concatenate the two vectors INLINEFORM2 and INLINEFORM3 into a bidirectional representation INLINEFORM4 . These vectors are further concatenated into the matrix INLINEFORM5 where the INLINEFORM6 th column corresponds to INLINEFORM7 .
Next, we generate the output one word at a time while referencing this encoded input sentence and tracking progress with a decoder LSTM. The decoder's hidden state INLINEFORM0 is a fixed-length continuous vector representing the previous target words INLINEFORM1 , initialized as INLINEFORM2 . Based on this INLINEFORM3 , we calculate a similarity vector INLINEFORM4 , with each element equal to DISPLAYFORM0
INLINEFORM0 can be an arbitrary similarity function, which we set to the dot product, following luong15emnlp. We then normalize this into an attention vector, which weights the amount of focus that we put on each word in the source sentence DISPLAYFORM0
This attention vector is then used to weight the encoded representation INLINEFORM0 to create a context vector INLINEFORM1 for the current time step INLINEFORM2
Finally, we create INLINEFORM0 by concatenating the previous hidden state INLINEFORM1 with the context vector, and performing an affine transform INLINEFORM2
Once we have this representation of the current state, we can calculate INLINEFORM0 according to Equation ( EQREF3 ). The next word INLINEFORM1 is chosen according to this probability, and we update the hidden state by inputting the chosen word into the decoder LSTM DISPLAYFORM0
If we define all the parameters in this model as INLINEFORM0 , we can then train the model by minimizing the negative log-likelihood of the training data INLINEFORM1
Integrating Lexicons into NMT
In § SECREF2 we described how traditional NMT models calculate the probability of the next target word INLINEFORM0 . Our goal in this paper is to improve the accuracy of this probability estimate by incorporating information from discrete probabilistic lexicons. We assume that we have a lexicon that, given a source word INLINEFORM1 , assigns a probability INLINEFORM2 to target word INLINEFORM3 . For a source word INLINEFORM4 , this probability will generally be non-zero for a small number of translation candidates, and zero for the majority of words in INLINEFORM5 . In this section, we first describe how we incorporate these probabilities into NMT, and explain how we actually obtain the INLINEFORM6 probabilities in § SECREF4 .
Converting Lexicon Probabilities into Conditioned Predictive Proabilities
First, we need to convert lexical probabilities INLINEFORM0 for the individual words in the source sentence INLINEFORM1 to a form that can be used together with INLINEFORM2 . Given input sentence INLINEFORM3 , we can construct a matrix in which each column corresponds to a word in the input sentence, each row corresponds to a word in the INLINEFORM4 , and the entry corresponds to the appropriate lexical probability: INLINEFORM5
This matrix can be precomputed during the encoding stage because it only requires information about the source sentence INLINEFORM0 .
Next we convert this matrix into a predictive probability over the next word: INLINEFORM0 . To do so we use the alignment probability INLINEFORM1 from Equation ( EQREF5 ) to weight each column of the INLINEFORM2 matrix: INLINEFORM3
This calculation is similar to the way how attentional models calculate the context vector INLINEFORM0 , but over a vector representing the probabilities of the target vocabulary, instead of the distributed representations of the source words. The process of involving INLINEFORM1 is important because at every time step INLINEFORM2 , the lexical probability INLINEFORM3 will be influenced by different source words.
Combining Predictive Probabilities
After calculating the lexicon predictive probability INLINEFORM0 , next we need to integrate this probability with the NMT model probability INLINEFORM1 . To do so, we examine two methods: (1) adding it as a bias, and (2) linear interpolation.
In our first bias method, we use INLINEFORM0 to bias the probability distribution calculated by the vanilla NMT model. Specifically, we add a small constant INLINEFORM1 to INLINEFORM2 , take the logarithm, and add this adjusted log probability to the input of the softmax as follows: INLINEFORM3
We take the logarithm of INLINEFORM0 so that the values will still be in the probability domain after the softmax is calculated, and add the hyper-parameter INLINEFORM1 to prevent zero probabilities from becoming INLINEFORM2 after taking the log. When INLINEFORM3 is small, the model will be more heavily biased towards using the lexicon, and when INLINEFORM4 is larger the lexicon probabilities will be given less weight. We use INLINEFORM5 for this paper.
We also attempt to incorporate the two probabilities through linear interpolation between the standard NMT probability model probability INLINEFORM0 and the lexicon probability INLINEFORM1 . We will call this the linear method, and define it as follows: INLINEFORM2
where INLINEFORM0 is an interpolation coefficient that is the result of the sigmoid function INLINEFORM1 . INLINEFORM2 is a learnable parameter, and the sigmoid function ensures that the final interpolation level falls between 0 and 1. We choose INLINEFORM3 ( INLINEFORM4 ) at the beginning of training.
This notation is partly inspired by allamanis16icml and gu16acl who use linear interpolation to merge a standard attentional model with a “copy” operator that copies a source word as-is into the target sentence. The main difference is that they use this to copy words into the output while our method uses it to influence the probabilities of all target words.
Constructing Lexicon Probabilities
In the previous section, we have defined some ways to use predictive probabilities INLINEFORM0 based on word-to-word lexical probabilities INLINEFORM1 . Next, we define three ways to construct these lexical probabilities using automatically learned lexicons, handmade lexicons, or a combination of both.
Automatically Learned Lexicons
In traditional SMT systems, lexical translation probabilities are generally learned directly from parallel data in an unsupervised fashion using a model such as the IBM models BIBREF7 , BIBREF8 . These models can be used to estimate the alignments and lexical translation probabilities INLINEFORM0 between the tokens of the two languages using the expectation maximization (EM) algorithm.
First in the expectation step, the algorithm estimates the expected count INLINEFORM0 . In the maximization step, lexical probabilities are calculated by dividing the expected count by all possible counts: INLINEFORM1
The IBM models vary in level of refinement, with Model 1 relying solely on these lexical probabilities, and latter IBM models (Models 2, 3, 4, 5) introducing more sophisticated models of fertility and relative alignment. Even though IBM models also occasionally have problems when dealing with the rare words (e.g. “garbage collecting” effects BIBREF9 ), traditional SMT systems generally achieve better translation accuracies of low-frequency words than NMT systems BIBREF6 , indicating that these problems are less prominent than they are in NMT.
Note that in many cases, NMT limits the target vocabulary BIBREF10 for training speed or memory constraints, resulting in rare words not being covered by the NMT vocabulary INLINEFORM0 . Accordingly, we allocate the remaining probability assigned by the lexicon to the unknown word symbol INLINEFORM1 : DISPLAYFORM0
Manual Lexicons
In addition, for many language pairs, broad-coverage handmade dictionaries exist, and it is desirable that we be able to use the information included in them as well. Unlike automatically learned lexicons, however, handmade dictionaries generally do not contain translation probabilities. To construct the probability INLINEFORM0 , we define the set of translations INLINEFORM1 existing in the dictionary for particular source word INLINEFORM2 , and assume a uniform distribution over these words: INLINEFORM3
Following Equation ( EQREF12 ), unknown source words will assign their probability mass to the INLINEFORM0 tag.
Hybrid Lexicons
Handmade lexicons have broad coverage of words but their probabilities might not be as accurate as the learned ones, particularly if the automatic lexicon is constructed on in-domain data. Thus, we also test a hybrid method where we use the handmade lexicons to complement the automatically learned lexicon. Specifically, inspired by phrase table fill-up used in PBMT systems BIBREF11 , we use the probability of the automatically learned lexicons INLINEFORM1 by default, and fall back to the handmade lexicons INLINEFORM2 only for uncovered words: DISPLAYFORM0
Experiment & Result
In this section, we describe experiments we use to evaluate our proposed methods.
Settings
Dataset: We perform experiments on two widely-used tasks for the English-to-Japanese language pair: KFTT BIBREF12 and BTEC BIBREF13 . KFTT is a collection of Wikipedia article about city of Kyoto and BTEC is a travel conversation corpus. BTEC is an easier translation task than KFTT, because KFTT covers a broader domain, has a larger vocabulary of rare words, and has relatively long sentences. The details of each corpus are depicted in Table TABREF19 .
We tokenize English according to the Penn Treebank standard BIBREF14 and lowercase, and tokenize Japanese using KyTea BIBREF15 . We limit training sentence length up to 50 in both experiments and keep the test data at the original length. We replace words of frequency less than a threshold INLINEFORM0 in both languages with the INLINEFORM1 symbol and exclude them from our vocabulary. We choose INLINEFORM2 for BTEC and INLINEFORM3 for KFTT, resulting in INLINEFORM4 k, INLINEFORM5 k for BTEC and INLINEFORM6 k, INLINEFORM7 k for KFTT.
NMT Systems: We build the described models using the Chainer toolkit. The depth of the stacking LSTM is INLINEFORM0 and hidden node size INLINEFORM1 . We concatenate the forward and backward encodings (resulting in a 1600 dimension vector) and then perform a linear transformation to 800 dimensions.
We train the system using the Adam BIBREF16 optimization method with the default settings: INLINEFORM0 . Additionally, we add dropout BIBREF17 with drop rate INLINEFORM1 at the last layer of each stacking LSTM unit to prevent overfitting. We use a batch size of INLINEFORM2 and we run a total of INLINEFORM3 iterations for all data sets. All of the experiments are conducted on a single GeForce GTX TITAN X GPU with a 12 GB memory cache.
At test time, we use beam search with beam size INLINEFORM0 . We follow luong15acl in replacing every unknown token at position INLINEFORM1 with the target token that maximizes the probability INLINEFORM2 . We choose source word INLINEFORM3 according to the highest alignment score in Equation ( EQREF5 ). This unknown word replacement is applied to both baseline and proposed systems. Finally, because NMT models tend to give higher probabilities to shorter sentences BIBREF18 , we discount the probability of INLINEFORM4 token by INLINEFORM5 to correct for this bias.
Traditional SMT Systems: We also prepare two traditional SMT systems for comparison: a PBMT system BIBREF19 using Moses BIBREF20 , and a hierarchical phrase-based MT system BIBREF21 using Travatar BIBREF22 , Systems are built using the default settings, with models trained on the training data, and weights tuned on the development data.
Lexicons: We use a total of 3 lexicons for the proposed method, and apply bias and linear method for all of them, totaling 6 experiments. The first lexicon (auto) is built on the training data using the automatically learned lexicon method of § SECREF11 separately for both the BTEC and KFTT experiments. Automatic alignment is performed using GIZA++ BIBREF8 . The second lexicon (man) is built using the popular English-Japanese dictionary Eijiro with the manual lexicon method of § SECREF13 . Eijiro contains 104K distinct word-to-word translation entries. The third lexicon (hyb) is built by combining the first and second lexicon with the hybrid method of § SECREF14 .
Evaluation: We use standard single reference BLEU-4 BIBREF23 to evaluate the translation performance. Additionally, we also use NIST BIBREF24 , which is a measure that puts a particular focus on low-frequency word strings, and thus is sensitive to the low-frequency words we are focusing on in this paper. We measure the statistical significant differences between systems using paired bootstrap resampling BIBREF25 with 10,000 iterations and measure statistical significance at the INLINEFORM0 and INLINEFORM1 levels.
Additionally, we also calculate the recall of rare words from the references. We define “rare words” as words that appear less than eight times in the target training corpus or references, and measure the percentage of time they are recovered by each translation system.
Effect of Integrating Lexicons
In this section, we first a detailed examination of the utility of the proposed bias method when used with the auto or hyb lexicons, which empirically gave the best results, and perform a comparison among the other lexicon integration methods in the following section. Table TABREF20 shows the results of these methods, along with the corresponding baselines.
First, compared to the baseline attn, our bias method achieved consistently higher scores on both test sets. In particular, the gains on the more difficult KFTT set are large, up to 2.3 BLEU, 0.44 NIST, and 30% Recall, demonstrating the utility of the proposed method in the face of more diverse content and fewer high-frequency words.
Compared to the traditional pbmt systems hiero, particularly on KFTT we can see that the proposed method allows the NMT system to exceed the traditional SMT methods in BLEU. This is despite the fact that we are not performing ensembling, which has proven to be essential to exceed traditional systems in several previous works BIBREF6 , BIBREF0 , BIBREF1 . Interestingly, despite gains in BLEU, the NMT methods still fall behind in NIST score on the KFTT data set, demonstrating that traditional SMT systems still tend to have a small advantage in translating lower-frequency words, despite the gains made by the proposed method.
In Table TABREF27 , we show some illustrative examples where the proposed method (auto-bias) was able to obtain a correct translation while the normal attentional model was not. The first example is a mistake in translating “extramarital affairs” into the Japanese equivalent of “soccer,” entirely changing the main topic of the sentence. This is typical of the errors that we have observed NMT systems make (the mistake from Figure FIGREF2 is also from attn, and was fixed by our proposed method). The second example demonstrates how these mistakes can then affect the process of choosing the remaining words, propagating the error through the whole sentence.
Next, we examine the effect of the proposed method on the training time for each neural MT method, drawing training curves for the KFTT data in Figure FIGREF26 . Here we can see that the proposed bias training methods achieve reasonable BLEU scores in the upper 10s even after the first iteration. In contrast, the baseline attn method has a BLEU score of around 5 after the first iteration, and takes significantly longer to approach values close to its maximal accuracy. This shows that by incorporating lexical probabilities, we can effectively bootstrap the learning of the NMT system, allowing it to approach an appropriate answer in a more timely fashion.
It is also interesting to examine the alignment vectors produced by the baseline and proposed methods, a visualization of which we show in Figure FIGREF29 . For this sentence, the outputs of both methods were both identical and correct, but we can see that the proposed method (right) placed sharper attention on the actual source word corresponding to content words in the target sentence. This trend of peakier attention distributions in the proposed method held throughout the corpus, with the per-word entropy of the attention vectors being 3.23 bits for auto-bias, compared with 3.81 bits for attn, indicating that the auto-bias method places more certainty in its attention decisions.
Comparison of Integration Methods
Finally, we perform a full comparison between the various methods for integrating lexicons into the translation process, with results shown in Table TABREF31 . In general the bias method improves accuracy for the auto and hyb lexicon, but is less effective for the man lexicon. This is likely due to the fact that the manual lexicon, despite having broad coverage, did not sufficiently cover target-domain words (coverage of unique words in the source vocabulary was 35.3% and 9.7% for BTEC and KFTT respectively).
Interestingly, the trend is reversed for the linear method, with it improving man systems, but causing decreases when using the auto and hyb lexicons. This indicates that the linear method is more suited for cases where the lexicon does not closely match the target domain, and plays a more complementary role. Compared to the log-linear modeling of bias, which strictly enforces constraints imposed by the lexicon distribution BIBREF27 , linear interpolation is intuitively more appropriate for integrating this type of complimentary information.
On the other hand, the performance of linear interpolation was generally lower than that of the bias method. One potential reason for this is the fact that we use a constant interpolation coefficient that was set fixed in every context. gu16acl have recently developed methods to use the context information from the decoder to calculate the different interpolation coefficients for every decoding step, and it is possible that introducing these methods would improve our results.
Additional Experiments
To test whether the proposed method is useful on larger data sets, we also performed follow-up experiments on the larger Japanese-English ASPEC dataset BIBREF28 that consist of 2 million training examples, 63 million tokens, and 81,000 vocabulary size. We gained an improvement in BLEU score from 20.82 using the attn baseline to 22.66 using the auto-bias proposed method. This experiment shows that our method scales to larger datasets.
Related Work
From the beginning of work on NMT, unknown words that do not exist in the system vocabulary have been focused on as a weakness of these systems. Early methods to handle these unknown words replaced them with appropriate words in the target vocabulary BIBREF10 , BIBREF29 according to a lexicon similar to the one used in this work. In contrast to our work, these only handle unknown words and do not incorporate information from the lexicon in the learning procedure.
There have also been other approaches that incorporate models that learn when to copy words as-is into the target language BIBREF30 , BIBREF31 , BIBREF32 . These models are similar to the linear approach of § UID10 , but are only applicable to words that can be copied as-is into the target language. In fact, these models can be thought of as a subclass of the proposed approach that use a lexicon that assigns a all its probability to target words that are the same as the source. On the other hand, while we are simply using a static interpolation coefficient INLINEFORM0 , these works generally have a more sophisticated method for choosing the interpolation between the standard and “copy” models. Incorporating these into our linear method is a promising avenue for future work.
In addition mi16acl have also recently proposed a similar approach by limiting the number of vocabulary being predicted by each batch or sentence. This vocabulary is made by considering the original HMM alignments gathered from the training corpus. Basically, this method is a specific version of our bias method that gives some of the vocabulary a bias of negative infinity and all other vocabulary a uniform distribution. Our method improves over this by considering actual translation probabilities, and also considering the attention vector when deciding how to combine these probabilities.
Finally, there have been a number of recent works that improve accuracy of low-frequency words using character-based translation models BIBREF33 , BIBREF34 , BIBREF35 . However, luong16acl have found that even when using character-based models, incorporating information about words allows for gains in translation accuracy, and it is likely that our lexicon-based method could result in improvements in these hybrid systems as well.
Conclusion & Future Work
In this paper, we have proposed a method to incorporate discrete probabilistic lexicons into NMT systems to solve the difficulties that NMT systems have demonstrated with low-frequency words. As a result, we achieved substantial increases in BLEU (2.0-2.3) and NIST (0.13-0.44) scores, and observed qualitative improvements in the translations of content words.
For future work, we are interested in conducting the experiments on larger-scale translation tasks. We also plan to do subjective evaluation, as we expect that improvements in content word translation are critical to subjective impressions of translation results. Finally, we are also interested in improvements to the linear method where INLINEFORM0 is calculated based on the context, instead of using a fixed value.
Acknowledgment
We thank Makoto Morishita and Yusuke Oda for their help in this project. We also thank the faculty members of AHC lab for their supports and suggestions.
This work was supported by grants from the Ministry of Education, Culture, Sport, Science, and Technology of Japan and in part by JSPS KAKENHI Grant Number 16H05873.

Question: What datasets were used?
Answer: KFTT BIBREF12 and BTEC BIBREF13

Question: What language pairs did they experiment with?
Answer: English-Japanese