On Search Strategies for Document-Level Neural Machine Translation
Year: 2023

Abstract:
Compared to sentence-level systems, document-level neural machine translation (NMT) models produce a more consistent output across a document and are able to better resolve ambiguities within the input. There are many works on document-level NMT, mostly focusing on modifying the model architecture or training strategy to better accommodate the additional context-input. On the other hand, in most works, the question on how to perform search with the trained model is scarcely discussed, sometimes not mentioned at all. In this work, we aim to answer the question how to best utilize a context-aware translation model in decoding. We start with the most popular document-level NMT approach and compare different decoding schemes, some from the literature and others proposed by us. In the comparison, we are using both, standard automatic metrics, as well as specific linguistic phenomena on three standard document-level translation benchmarks. We find that most commonly used decoding strategies perform similar to each other and that higher quality context information has the potential to further improve the translation.

heading: 1 Introduction
text: Neural machine translation (NMT) (Bahdanau et al., 2014; Vaswani et al., 2017) is widely adopted and produces excellent translations for many domains and language pairs. However, when these automatic translations are evaluated on the document level, they reveal shortcomings when it comes to consistency in style, entity-translation or correct inference of the gender, among other things (Läubli et al., 2018; Müller et al., 2018; Thai et al., 2022). Document-level NMT aims to resolve these shortcomings by taking the context of a sentence into account during translation. There exist many works on the topic of document-level NMT, proposing various changes to the standard transformer (Vaswani
et al., 2017) architecture and training criteria to improve context incorporation and consequently translation quality. However, while the modeling and training aspects are covered in great detail in these works, the exact decoding strategy is often not very clearly described and sometimes not mentioned at all.
In this work, we head out to answer the question, which decoding strategy is most beneficial for document-level NMT systems. We compare all commonly used strategies, as well as some additional ones, on three standard document-level translation benchmarks. We find that most of the analyzed decoding strategies perform similar to each other. Also, higher quality context information can lead to better translations in certain scenarios.

heading: 2 Related Work
text: The earliest approaches to document-level NMT simply concatenate consecutive sentences without any further changes to the architecture compared to the sentence-level systems (Tiedemann and Scherrer, 2017; Agrawal et al., 2018). Later, some changes were made to the vanilla transformer architecture, like segment embeddings (Ma et al., 2020) or attention masking (Zhang et al., 2020; Petrick et al., 2022) and a move was made towards translating longer segments (Junczys-Dowmunt, 2019; Liu et al., 2020; Zheng et al., 2021; Bao et al., 2021; Sun et al., 2022). Other works employ a separate encoder to include the additional context on the source side (Jean et al., 2017; Bawden et al., 2018; Zhang et al., 2018; Voita et al., 2018) or make use of the context in a post-editing fashion (Voita et al., 2019; Xiong et al., 2019). Further approaches include the usage of a cache (Wang et al., 2017; Maruf and Haffari, 2018; Tu et al., 2018) or hierarchical attention networks (Miculicich et al., 2018; Maruf et al., 2019; Wong et al., 2020). Recently, several works have concluded that the simple concatenation approach used with the
12827
vanilla transformer architecture performs as good - if not better - than more complicated approaches that modify the model structure (Sun et al., 2022; Majumde et al., 2022). Since we also observed this in our internal comparisons, we decided to focus on this simple approach for our analysis in this work.
Several works made the argument that the improvements seen in automatic metric scores for document-level NMT systems are from regularization effects rather than from utilizing the additional context information (Kim et al., 2019; Li et al., 2020; Nguyen et al., 2021). In order to better asses the improvements gained by documentlevel NMT, several targeted test suites have been released (Müller et al., 2018; Bawden et al., 2018; Voita et al., 2019; Jwalapuram et al., 2019). However, all of these are based on just scoring contrastive examples without actually translating anything. Recently, Jiang et al. (2022) and Currey et al. (2022) have released frameworks that allow to score MT systems on their ability to generate contextually correct translations.1

heading: 3 Search Strategies
text: Training a document-level NMT system that takes the last k sentences as context is straightforward using the standard concatenation strategy (Tiedemann and Scherrer, 2017). Given some document level training data (Fn, En), n = 1, ..., N , where (Fn, En) denotes the n-th source-target sentence pair, during training we optimize the parameters Θ of the model towards
Θ̂ = argmax Θ
{∑
n
log pΘ(E n n−k|Fnn−k)
} .
Here, Enn−k denotes the concatenation of the sentences En−k, ..., En.
During search, given a document FM1 , we want to find the best translation ÊM1 according to the model. Of course, exact search can not be performed and different works have used different methods to generate a translation:
full segment (Liu et al., 2020; Bao et al., 2021; Sun et al., 2022): we split the document into non-overlapping parts F k1 , F 2k k+1, ..., F M M−k
and translate each part separately using
Êii−k = argmax Eii−k
{ p(Eii−k|F ii−k) } , (1)
1The framework by Currey et al. (2022) was not yet made available when we conducted our experiments.
which is approximated using standard beam search on the token level.
last sentence (Bawden et al., 2018; Agrawal et al., 2018; Zhang et al., 2020; Petrick et al., 2022; Majumde et al., 2022): we split the document into overlapping parts ..., F ii−k, F i+1 i−k+1, ...
and translate each part separately using Equation 1. From each translated part we choose only the last sentence to get one translation for every sentence in the document.
first sentence (Zhang et al., 2020): similar to last sentence, but from each translated part we choose only the first sentence to get one translation for every sentence in the document.
2-pass decoding (Maruf and Haffari, 2018; Maruf et al., 2019; Voita et al., 2019; Xiong et al., 2019): we first generate a translation ẼM1 of the document using a sentence-level NMT system. Then, the final hypothesis Êi for each sentence Fi is created using
Êi = argmax Ei
{ p(Ei|F ii−k, Ẽi−1i−k) } .
doc-trans (Miculicich et al., 2018; Voita et al., 2019; Garcia et al., 2019; Fernandes et al., 2021): we generate the translation sentence by sentence, meaning
Ê1 = argmax E1
{p(E1|F1)} ,
Ê2 = argmax E2
{ p(E2|F 21 , Ê1) } ,
...
doc-trans (beam) : similar to doc-trans, but instead of keeping just the best context Êi−11 , we keep the top-h candidates and prune them after each step i, analogous to beam search on the token level. h = 12 for all our experiments, the same as our token-level beam-size.
cheating : this is just used as a tool for analysis. The translation of each sentence Fi is created using the true target reference E̊M1 as context
Êi = argmax Ei
{ p(Ei|F ii−k, E̊i−1i−k) } .
no context : this is just used as a tool for analysis. The translation of each sentence Fi is created using no context information at all
Êi = argmax Ei
{p(Ei|Fi)} .
The different search strategies also have a different computational cost associated with them. The biggest factor regarding the decoding cost is the number of forward passes through the model, specifically the decoder, that we have to do. We list the computational costs for the different decoding approaches in Table 1 under the assumption that the document consists of N sentences with average sentence length L and the model uses k − 1 sentences as context. Please note that the decoding time might follow a different dependence than the cost in the above table, since it heavily depends on the available hardware. For example, doc trans and doc trans (beam) might have the same decoding time, if we have enough computational resources available, since the additional computations in doc trans (beam) can all be done in parallel.

heading: 4 Experiments
text: We perform experiments on three document-level translation benchmarks, called NEWS (En→De), TED (En→It) and OS (En→De). For the details regarding data conditions and preparation, as well as model training, we refer to Appendix A. For the context-aware systems, we concatenate 3 adjacent sentences (i.e. k = 3) using a special token <sep>. For the two En→De tasks, we also evaluate the systems on the ContraPro test set (Müller et al., 2018). Instead of scoring and ranking the contrastive examples in ContraPro, as the authors have originally envisioned, we translate the source side to calculate BLEU and TER as well as to score the pronoun translations according to Section 4.1. We can not evaluate the full segment search strategy on ContraPro, because the sentences are not adjacent since they come from different documents.

heading: 4.1 Evaluating Pronoun Translation
text: As further analysis, we measure how well ambiguous pronouns are handled when translating from English to German. Regarding gender, the English third-person pronoun ‘it’ (and its other forms), can be translated to the German words ‘er’, ‘sie’ or ‘es’, depending on which noun it refers to. On the other hand, ambiguities in the formality come from second-person pronouns. For example, the English word ‘you’can be translated to ‘sie’ or ‘du’ depending if we are in a formal setting or not. To report accuracies for pronoun (3 classes: male/female/neuter) and formality (2 classes: formal/informal) translation, we extend the BlonDe metric created by Jiang et al. (2022)2. First, we expand the framework to work for German references, by including German NER and POS taggers3 as well as including German pronoun mappings. For the gender category, we mostly follow Jiang et al. (2022), but additionally require that a corresponding pronoun must also be present in the source sentence.4 For the style category, we take into account examples where a second person pronoun appears in the source sentence, and a corresponding formal
2Our extension can be found in this fork: https:// github.com/christian3141/BlonDe
3https://spacy.io/models/de 4In ContraPro, we find 5011/4085/4817 examples for male/female/neuter respectively. The difference to the 4000/4000/4000 reported by (Müller et al., 2018) means that in some cases we count multiple occurrences in a single example.
or informal pronoun appears in the reference.5

heading: 4.2 Perplexities
text: First, we compare the perplexities of the hypotheses from the different search strategies, which are listed in Table 2. The first thing to note is, that the reference has a much higher perplexity than all hypotheses, which is commonly seen for NMT systems. All document-level search strategies result in different hypotheses, which however have a similar perplexity score. Surprisingly, the cheating setting generates the worst translation perplexity-
5In the OS test set, we count 416 and 605 examples for formal and informal examples respectively.
wise, even worse than using no context. This might be related to the observation, that the reference has a worse perplexity than any hypothesis, which is rather a modelling error than a search error.

heading: 4.3 Automatic Metrics
text: Next, we evaluate the hypotheses based on the common automatic metrics BLEU and TER. The results are shown in Table 3. The hypotheses created with no context seem to have the same quality as the sentence-level baseline. Surprisingly, the true reference as context does not improve performance on the NEWS and TED test sets. This indicates that the improvements seen on these test sets for the document-level system might not be related to better context incorporation. On the contrary, the OS system creates the best hypothesis with the true reference as context. All the actual decoding strategies give similar performance in terms of BLEU and TER with 2-pass decoding being a little bit behind. A special case is the first sentence strategy, which performs quite well on the standard test sets but poorly on ContraPro. This is, because ContraPro is designed in a way that the left side context is more important for translation than the right side.
Finally, we analyze the quality of the pronoun translation as discussed in Section 4.1. In principle, we could calculate the F1 score for both, gender and formality, on all En→De test sets. However, we discard the cases where one or more classes have less than 100 examples. This leaves us with the three test sets depicted in Table 4. As a sanity check,
we also report the ContraPro accuracies calculated from scoring the contrastive references as described in (Müller et al., 2018). They are 48.2/45.8 for sentence-level and 68.2/82.2 for document-level for NEWS/OS respectively. That means, with just scoring, we overestimate the capabilities of the system, but the trend is still consistent.6 Using the true reference leads to the best results in all cases. no context and first sentence leaves us with sentencelevel performance on the gender tasks, while all other decoding strategies perform similarly. For the formality, none of the methods can significantly outperform the sentence level system, although the cheating experiment shows that the system could do better if a better context information is provided. This might be, because segments of 3 sentences are too short to reliably detect if a setting is formal or informal, without access to the true reference.

heading: 5 Conclusion
text: In this work, we analyze decoding strategies for document-level NMT systems. Using the most popular document-level translation approach, we compare different search strategies found in the literature against methods developed by us. We find that most of the commonly used decoding strategies result in similar performance, both in terms of common automatic metrics, as well as on specific pronoun evaluation tasks. Therefore, we conclude that it is important to include the context information during decoding, but the exact way in which to do this is not as important. Also, we find that the document-level systems could actually profit from higher quality context information, in situations where this context is most relevant for translation.

heading: Acknowledgements
text: This work was partially supported by the project HYKIST funded by the German Federal Ministry of Health on the basis of a decision of the German Federal Parliament (Bundestag) under funding ID ZMVI1-2520DAT04A, and by NeuroSys which, as part of the initiative “Clusters4Future”, is funded by the Federal Ministry of Education and Research BMBF (03ZU1106DA).
Limitations
In this work, we limit our experiments to the most commonly used document-level system architec-
6The precision and recall are roughly the same for the F1 scores reported in Table 4.
ture and training criterion. Other approaches exist, which might exhibit a different behavior in decoding. Two out of the three document-level translation tasks we use in this work are low resource with less than 500k sentence-pairs as training data. We chose these tasks due to computational limitations and to be better comparable to other works, but higher resource scenarios are more realistic for actual applications. We limit the analysis of pronoun translation to the English-German language pair. Also, there are other aspects of documentlevel NMT, like consistent translation of entities, which we did not consider in our analysis.

heading: A Appendix
text: For the NEWS En→De task, the parallel training data (around 300k sentence pairs, newsdomain) comes from the NewsCommentaryV14 corpus7. As validation/test set we use the WMT newstest2015/newstest2018 test sets from the WMT news translation tasks (Farhad et al., 2021). For the TED En→It task, the parallel training data (around 200k sentence pairs, scientific-talksdomain) comes from the IWSLT17 Multilingual Task (Cettolo et al., 2017). As validation set we use the concatenation of IWSLT17.TED.dev2010 and IWSLT17.TED.tst2010 and as test set we use IWSLT17.TED.tst2017.mltlng. For the OS En→De task, the parallel training data (around 22.5M sentence pairs, subtitle-domain) comes from the OpenSubtitlesV2018 corpus (Lison et al., 2018). We use the same train/validation/test splits as Huo et al. (2020) and additionally remove all segments that are used in the ContraPro test suite (Müller et al., 2018) from the training data. The data statistics for all tasks can be found in Table 5.
Since in the original release of ContraPro only left side context is provided, we extract the right side context ourselves from OpenSubtitlesV2018 based on the meta-information of the segments.
We tokenize the data using byte-pair-encoding (Sennrich et al., 2016; Kudo, 2018) with 15k joint merge operations (32k for OS En→De). The models are implemented using the fairseq toolkit (Ott et al., 2019) following the transformer base architecture (Vaswani et al., 2017) with dropout 0.3 and label-smoothing 0.2 for NEWS En→De and TED
7https://data.statmt.org/news-commentary/v14/
En→It and dropout 0.1 and label-smoothing 0.1 for OS En→De. This resulted in models with ca. 51M parameters for NEWS and TED and ca. 60M parameters for OS for both the sentencelevel and the document-level systems. All systems are trained until the validation perplexity does no longer improve and the best checkpoint is selected using validation perplexity as well. Training took around 24h for NEWS and TED and around 96h for OS on a single NVIDIA GeForce RTX 2080 Ti graphics card. Due to computational limitations, we report results only for a single run. For the generation of segments (see Section 3), we use beam-search on the token level with beam-size 12 and length normalization. To calculate BLEU (Papineni et al., 2002) and TER (Snover et al., 2006) we use SacreBLEU (Post, 2018).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitations
7 A2. Did you discuss any potential risks of your work? The authors do not foresee potential risks of this work.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4 Experiments

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4 Experiments
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? All artifacts that were used allow such usage for research purposes.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? All artifacts that were used allow such usage for research purposes.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We only use standard datasets which allow usage for research purposes.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4 Experiments
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section Appendix
C 3 Did you run computational experiments? Section 4 Experiments

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4 Experiments

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section Appendix

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4 Experiments
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: This work was partially supported by the project HYKIST funded by the German Federal Ministry of Health on the basis of a decision of the German Federal Parliament (Bundestag) under funding ID ZMVI1-2520DAT04A, and by NeuroSys which, as part of the initiative “Clusters4Future”, is funded by the Federal Ministry of Education and Research BMBF (03ZU1106DA). Limitations
In this work, we limit our experiments to the most commonly used document-level system architec-
6The precision and recall are roughly the same for the F1 scores reported in Table 4. ture and training criterion. Other approaches exist, which might exhibit a different behavior in decoding. Two out of the three document-level translation tasks we use in this work are low resource with less than 500k sentence-pairs as training data. We chose these tasks due to computational limitations and to be better comparable to other works, but higher resource scenarios are more realistic for actual applications. We limit the analysis of pronoun translation to the English-German language pair. Also, there are other aspects of documentlevel NMT, like consistent translation of entities, which we did not consider in our analysis