Using Whole Document Context in Neural Machine Translation

Abstract: In Machine Translation, considering the document as a whole can help to resolve ambiguities and inconsistencies. In this paper, we propose a simple yet promising approach to add contextual information in Neural Machine Translation. We present a method to add source context that capture the whole document with accurate boundaries, taking every word into account. We provide this additional information to a Transformer model and study the impact of our method on three language pairs. The proposed approach obtains promising results in the English-German, English-French and French-English document-level translation tasks. We observe interesting cross-sentential behaviors where the model learns to use document-level information to improve translation coherence.

Neural machine translation (NMT) has grown rapidly in the past years BIBREF0, BIBREF1. It usually takes the form of an encoder-decoder neural network architecture in which source sentences are summarized into a vector representation by the encoder and are then decoded into target sentences by the decoder. NMT has outperformed conventional statistical machine translation (SMT) by a significant margin over the past years, benefiting from gating and attention techniques. Various models have been proposed based on different architectures such as RNN BIBREF0, CNN BIBREF2 and Transformer BIBREF1, the latter having achieved state-of-the-art performances while significantly reducing training time.
However, by considering sentence pairs separately and ignoring broader context, these models suffer from the lack of valuable contextual information, sometimes leading to inconsistency in a translated document. Adding document-level context helps to improve translation of context-dependent parts. Previous study BIBREF3 showed that such context gives substantial improvement in the handling of discourse phenomena like lexical disambiguation or co-reference resolution.
Most document-level NMT approaches focus on adding contextual information by taking into account a set of sentences surrounding the current pair BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9. While giving significant improvement over the context-agnostic versions, none of these studies consider the whole document with well delimited boundaries. The majority of these approaches also rely on structural modification of the NMT model BIBREF6, BIBREF7, BIBREF8, BIBREF9. To the best of our knowledge, there is no existing work considering whole documents without structural modifications.
Contribution: We propose a preliminary study of a generic approach allowing any model to benefit from document-level information while translating sentence pairs. The core idea is to augment source data by adding document information to each sentence of a source corpus. This document information corresponds to the belonging document of a sentence and is computed prior to training, it takes every document word into account. Our approach focuses on pre-processing and consider whole documents as long as they have defined boundaries. We conduct experiments using the Transformer base model BIBREF1. For the English-German language pair we use the full WMT 2019 parallel dataset. For the English-French language pair we use a restricted dataset containing the full TED corpus from MUST-C BIBREF10 and sampled sentences from WMT 2019 dataset. We obtain important improvements over the baseline and present evidences that this approach helps to resolve cross-sentence ambiguities.
Interest in considering the whole document instead of a set of sentences preceding the current pair lies in the necessity for a human translator to account for broader context in order to keep a coherent translation. The idea of representing and using documents for a model is interesting, since the model could benefit from information located before or after the current processed sentence.
Previous work on document-level SMT started with cache based approaches, BIBREF11 suggest a conjunction of dynamic, static and topic-centered cache. More recent work tend to focus on strategies to capture context at the encoder level. Authors of BIBREF5 propose an auxiliary context source with a RNN dedicated to encode contextual information in addition to a warm-start of encoder and decoder states. They obtain significant gains over the baseline.
A first extension to attention-based neural architectures is proposed by BIBREF6, they add an encoder devoted to capture the preceding source sentence. Authors of BIBREF7 introduce a hierarchical attention network to model contextual information from previous sentences. Here the attention allows dynamic access to the context by focusing on different sentences and words. They show significant improvements over a strong NMT baseline. More recently, BIBREF9 extend Transformer architecture with an additional encoder to capture context and selectively merge sentence and context representations. They focus on co-reference resolution and obtain improvements in overall performances.
The closest approach to ours is presented by BIBREF4, they simply concatenate the previous source sentence to the one being translated. While they do not make any structural modification to the model, their method still does not take the whole document into account.
We propose to use the simplest method to estimate document embeddings. The approach is called SWEM-aver (Simple Word Embedding Model – average) BIBREF12. The embedding of a document $k$ is computed by taking the average of all its $N$ word vectors (see Eq. DISPLAY_FORM2) and therefore has the same dimension. Out of vocabulary words are ignored.
Despite being straightforward, our approach raises the need of already computed word vectors to keep consistency between word and document embeddings. Otherwise, fine-tuning embeddings as the model is training would shift them in a way that totally wipes off the connection between document and word vectors.
To address this problem, we adopt the following approach: First, we train a baseline Transformer model (noted Baseline model) from which we extract word embeddings. Then, we estimate document embeddings using the SWEM-aver method and train an enhanced model (noted Document model) benefiting from these document embeddings and the extracted word embeddings. During training, the Document model does not fine-tune its embeddings to preserve the relation between words and document vectors. It should be noted that we could directly use word embeddings extracted from another model such as Word2Vec BIBREF13, in practice we obtain better results when we get these vectors from a Transformer model. In our case, we simply extract them from the Baseline after it has been trained.
Using domain adaptation ideas BIBREF14, BIBREF15, BIBREF16, we associate a tag to each sentence of the source corpus, which represents the document information. This tag takes the form of an additional token placed at the first position in the sentence and corresponds to the belonging document of the sentence (see Table TABREF1). The model considers the tag as an additional word and replace it with the corresponding document embedding. The Baseline model is trained on a standard corpus that does not contain document tags, while the Document model is trained on corpus that contains document tags.
The proposed approach requires strong hypotheses about train and test data. The first downfall is the need for well defined document boundaries that allow to mark each sentence with its document tag. The second major downfall is the need to compute an embedding vector for each new document fed in the model, adding a preprocessing step before inference time.
We consider two different models for each language pair: the Baseline and the Document model. We evaluate them on 3 test sets and report BLEU and TER scores. All experiments are run 8 times with different seeds, we report averaged results and p-values for each experiment.
Translation tasks are English to German, proposed in the first document-level translation task at WMT 2019 BIBREF17, English to French and French to English, following the IWSLT translation task BIBREF18.
Table TABREF4 describes the data used for the English-German language pair. These corpora correspond to the WMT 2019 document-level translation task. Table TABREF5 describes corpora for the English-French language pair, the same data is used for both translation directions.
For the English-German pair, only 10.4% (3.638M lines) of training data contains document boundaries. For English-French pair, we restricted the total amount of training data in order to keep 16.1% (602K lines) of document delimited corpora. To achieve this we randomly sampled 10% of the ParaCrawl V3. It means that only a fraction of the source training data contains document context. The enhanced model learns to use document information only when it is available.
All test sets contain well delimited documents, Baseline models are evaluated on standard corpora while Document models are evaluated on the same standard corpora that have been augmented with document context. We evaluate the English-German systems on newstest2017, newstest2018 and newstest2019 where documents consist of newspaper articles to keep consistency with the training data. English to French and French to English systems are evaluated over IWSLT TED tst2013, tst2014 and tst2015 where documents are transcriptions of TED conferences (see Table TABREF5).
Prior to experiments, corpora are tokenized using Moses tokenizer BIBREF19. To limit vocabulary size, we adopt the BPE subword unit approach BIBREF20, through the SentencePiece toolkit BIBREF21, with 32K rules.
We use the OpenNMT framework BIBREF22 in its TensorFlow version to create and train our models. All experiments are run on a single NVIDIA V100 GPU. Since the proposed approach relies on a preprocessing step and not on structural enhancement of the model, we keep the same Transformer architecture in all experiments. Our Transformer configuration is similar to the baseline of BIBREF1 except for the size of word and document vectors that we set to $d_{model} = 1024$, these vectors are fixed during training. We use $N = 6$ as the number of encoder layers, $d_{ff} = 2048$ as the inner-layer dimensionality, $h = 8$ attention heads, $d_k = 64$ as queries and keys dimension and $Pdrop = 0.1$ as dropout probability. All experiments, including baselines, are run over 600k training steps with a batch size of approximately 3000 tokens.
For all language pairs we trained a Baseline and a Document model. The Baseline is trained on a standard parallel corpus and is not aware of document embeddings, it is blind to the context and cannot link the sentences of a document. The Document model uses extracted word embeddings from the Baseline as initialization for its word vectors and also benefits from document embeddings that are computed from the extracted word embeddings. It is trained on the same corpus as the Baseline one, but the training corpus is augmented with (see Table TABREF1) and learns to make use of the document context.
The Document model does not consider its embeddings as tunable parameters, we hypothesize that fine-tuning word and document vectors breaks the relation between them, leading to poorer results. We provide evidence of this phenomena with an additional system for the French-English language pair, noted Document+tuning (see Table TABREF7) that is identical to the Document model except that it adjusts its embeddings during training.
The evaluated models are obtained by taking the average of their last 6 checkpoints, which were written at 5000 steps intervals. All experiments are run 8 times with different seeds to ensure the statistical robustness of our results. We provide p-values that indicate the probability of observing similar or more extreme results if the Document model is actually not superior to the Baseline.
Table TABREF6 presents results associated to the experiments for the English to German translation task, models are evaluated on the newstest2017, neswtest2018 and newstest2019 test sets. Table TABREF7 contains results for both English to French and French to English translation tasks, models are evaluated on the tst2013, tst2014 and tst2015 test sets.
En$\rightarrow $De: The Baseline model obtained State-of-The-Art BLEU and TER results according to BIBREF23, BIBREF24. The Document system shows best results, up to 0.85 BLEU points over the Baseline on the newstest2019 corpus. It also surpassed the Baselinee by 0.18 points on the newstest2017 with strong statistical significance, and by 0.15 BLEU points on the newstest2018 but this time with no statistical evidence. These encouraging results prompted us to extend experiments to another language pair: English-French.
En$\rightarrow $Fr: The Document system obtained the best results considering all metrics on all test sets with strong statistical evidence. It surpassed the Baseline by 1.09 BLEU points and 0.85 TER points on tst2015, 0.75 BLEU points and 0.76 TER points on tst2014, and 0.48 BLEU points and 0.68 TER points on tst2013.
Fr$\rightarrow $En: Of all experiments, this language pair shows the most important improvements over the Baseline. The Document model obtained substantial gains with very strong statistical evidence on all test sets. It surpassed the Baseline model by 1.81 BLEU points and 1.02 TER points on tst2015, 1.50 BLEU points and 0.96 TER points on tst2014, and 1.29 BLEU points and 0.83 TER points on tst2013.
The Document+tuning system, which only differs from the fact that it tunes its embeddings, shows little or no improvement over the Baseline, leading us to the conclusion that the relation between word and document embeddings described by Eq. DISPLAY_FORM2 must be preserved for the model to fully benefit from document context.
In this analysis we present some of the many cases that suggest the Document model can handle ambiguous situations. These examples are often isolated sentences where even a human translator could not predict the good translation without looking at the document, making it almost impossible for the Baseline model which is blind to the context. Table TABREF10 contains an extract of these interesting cases for the French-English language pair.
Translation from French to English is challenging and often requires to take the context into account. The personal pronoun "lui" can refer to a person of feminine gender, masculine gender or even an object and can therefore be translated into "her", "him" or "it". The first example in Table TABREF10 perfectly illustrate this ambiguity: the context clearly indicates that "lui" in the source sentence refers to "ma fille", which is located three sentences above, and should be translated into "her". In this case, the Baseline model predict the personal pronoun "him" while the Document model correctly predicts "her". It seems that the Baseline model does not benefit from any valuable information in the source sentence. Some might argue that the source sentence actually contains clues about the correct translation, considering that "robe à paillettes" ("sparkly dress") and "baguette magique" ("magic wand") probably refer to a little girl, but we will see that the model makes similar choices in more restricted contexts. This example is relevant mainly because the actual reference to the subject "ma fille" is made long before the source sentence.
The second example in Table TABREF10 is interesting because none of our models correctly translate the source sentence. However, we observe that the Baseline model opts for a literal translation of "je peux faire le poirier" ("I can stand on my head") into "I can do the pear" while the Document model predicts "I can wring". Even though these translations are both incorrect, we observe that the Document model makes a prediction that somehow relates to the context: a woman talking about her past disability, who has become more flexible thanks to yoga and can now twist her body.
The third case in table TABREF10 is a perfect example of isolated sentence that cannot be translated correctly with no contextual information. This example is tricky because the word "Elle" would be translated into "She" in most cases if no additional information were provided, but here it refers to "la conscience" ("consciousness") from the previous sentence and must be translated into "It". As expected the Baseline model does not make the correct guess and predicts the personal pronoun "She" while the Document model correctly predicts "It". This example present a second difficult part, the word "son" from the source sentence is ambiguous and does not, in itself, inform the translator if it must be translated into "her", "his" or "its". With contextual information we know that it refers to "[le] monde physique" ("[the] physical world") and that the correct choice is the word "its". Here the Baseline incorrectly predicts "her", possibly because of its earlier choice for "She" as the subject. The Document model makes again the correct translation.
According to our results (see Table TABREF7), the English-French language pair also benefits from document-level information but to a lesser extent. For this language pair, ambiguities about personal pronouns are less frequent. Other ambiguous phenomena like the formal mode (use of "vous" instead of "tu") appear. TableTABREF11 presents an example of this kind of situation where the word "You" from the source sentence does not indicate if the correct translation is "Vous" or "Tu". However it refers to the narrator of the story who is an old police officer. In this case, it is very likely that the use of formal mode is the correct translation. The Baseline model incorrectly predicts "Tu" and the Document model predicts "Vous".
In this work, we presented a preliminary study of a simple approach for document-level translation. The method allows to benefit from the whole document context at the sentence level, leading to encouraging results. In our experimental setup, we observed improvement of translation outcomes up to 0.85 BLEU points in the English to German translation task and exceeding 1 BLEU point in the English to French and French to English translation tasks. Looking at the translation outputs, we provided evidence that the approach allows NMT models to disambiguate complex situations where the context is absolutely necessary, even for a human translator.
The next step is to go further by investigating more elaborate document embedding approaches and to bring these experiments to other languages (e.g.: Asian, Arabic, Italian, Spanish, etc.). To consider a training corpus with a majority of document delimited data is also very promising.

Figure and table captions:
Table 1: Example of augmented parallel data used to train theDocumentmodel. The source corpus contains document tags while the target corpus remains unchanged.
Table 2: Detail of training and evaluation sets for the English-German pair, showing the number of lines, words in English (EN) and words in German (DE). Corpora with document boundaries are denoted by †.
Table 3: Detail of training and evaluation sets for the English-French pair in both directions, showing the number of lines, words in English (EN) and words in French (FR). Corpora with document boundaries are denoted by †.
Table 4: Results obtained for the English-German translation task, scored on three test sets using BLEU and TER metrics. p-values are denoted by * and correspond to the following values: ∗< .05, ∗∗< .01, ∗∗∗< .001.
Table 5: Results obtained for the English-French and French-English translation tasks, scored on three test sets using BLEU and TER metrics. p-values are denoted by * and correspond to the following values: ∗< .05, ∗∗< .01, ∗∗∗< .001.
Table 6: Translation examples for the French-English pair. We took the best models of all runs for both the Baseline and the Document enhanced model
Table 7: Translation example for the English-French pair.

Questions about this paper:
Which language-pair had the better performance?

Answers:
French-English