Venue: ACL 2023

# When Does Translation Require Context?

A Data-driven, Multilingual Exploration

Patrick Fernandes\({}^{1,2,3}\)+ **Kayo Yin\({}^{4}\)+ **Emmy Liu\({}^{1}\)**

&Andre F. T. Martins\({}^{2,3,5}\)  **Graham Neubig\({}^{1}\)**

\({}^{1}\)Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA

\({}^{2}\)Instituto Superior Tecnico & LUMLIS (Lisbon ELLIS Unit), Lisbon, Portugal

\({}^{3}\)Instituto de Telecomunicacoes, Lisbon, Portugal

\({}^{4}\)University of California, Berkeley

\({}^{5}\)Unbabel, Lisbon, Portugal

pfernand@cs.cmu.edu kayoyin@berkeley.edu

 Equal contributionCode available at [https://github.com/CoderPat/MuDA](https://github.com/CoderPat/MuDA) . See SSA for example usages of our released toolkit

###### Abstract

Although proper handling of discourse significantly contributes to the quality of machine translation (MT), these improvements are not adequately measured in common translation quality metrics. Recent works in context-aware MT attempt to target a small set of discourse phenomena during evaluation, however not in a fully systematic way. In this paper, we develop the **M**ultilingual **D**iscourse-**A**ware (MuDA) benchmark, a series of taggers that identify and evaluate model performance on discourse phenomena in any given dataset. The choice of phenomena is inspired by a novel methodology to systematically identify translations requiring context. We confirm the difficulty of previously studied phenomena while uncovering others that were previously unaddressed. We find that common context-aware MT models make only marginal improvements over context-agnostic models, which suggests these models do not handle these ambiguities effectively. We release code and data for 14 language pairs to encourage the MT community to focus on accurately capturing discourse phenomena.1

Footnote 1: Code available at [https://github.com/CoderPat/MuDA](https://github.com/CoderPat/MuDA) . See §A for example usages of our released toolkit

## 1 Introduction

In order to properly translate discourse phenomena including anaphoric pronouns, lexical cohesion, and discourse markers, a machine translation (MT) model must use information from previous utterances Guillou et al. (2018); Laubli et al. (2018); Toral et al. (2018).

However, while generating proper translations of these phenomena is important for comprehension, they represent a small portion of words in natural language. Therefore, common metrics such as BLEU Papinieni et al. (2002) cannot be used to judge the quality of discourse translation.

Recent work on neural machine translation (NMT) models that attempt to incorporate extrasentential context Tiedemann and Scherrer (2017); Miculicich et al. (2018); Maruf and Haffari (2018)_inter alia_) often perform targeted evaluation of certain discourse phenomena, mostly focusing on ellipsis, formality Voita et al. (2019, 2019), and pronoun translation Muller et al. (2018); Bawden et al. (2018); Lopes et al. (2020). However, only a limited set of discourse phenomena for a few language pairs have been studied (see summary in Table 1). The difficulty of broadening these studies stems from the reliance of previous work on introspection and domain knowledge to identify the relevant discourse phenomena, frequently involving expert speakers, which then requires engineering complex language-specific methods to create test suites or manually designing data for evaluation.

In this paper, we identify sentences that contain discourse phenomena through a _data-driven, semi-automatic methodology_. We apply this method to create a _multilingual benchmark testing discourse phenomena_ in the domain of MT. First, we develop P-CXMI (SS2) as a metric to identify when context is helpful in MT, or more broadly text generation in general. Then, we perform a systematic analysis of words with high P-CXMI to find categories of trans

\begin{table}
\begin{tabular}{p{65.4pt}|p{56.9pt}|p{56.9pt}} \hline Dataset & Lang. & Phenomena \\ \hline Müller et al. (2018) & EN \(\rightarrow\) DE & Pronouns \\ \hline Bowden et al. (2018) & EN \(\rightarrow\) FR & Pronouns, Coherence Lexical Consistency \\ \hline Voita et al. (2018) & EN \(\rightarrow\) RU & Prouns \\ Voita et al. (2019) & EN \(\rightarrow\) RU & Deixis, Ellipipsis \\  & Lexical Consistency \\ \hline Jwaluparam et al. (2020) & DE \(\rightarrow\) EN & Pronouns, Coherence Lexical Consistency Discourse Conectives \\ \hline Our Work & 14 Pairs (§S) & Pronouns, Ellipits Formalized Consistently Verh Forms \\ \hline \end{tabular}
\end{table}
Table 1: Some representative works on contextual machine translation that perform evaluation on discourse phenomena, contrasted to our work. For a more complete review see Maruf et al. (2021).



lations where context is useful (SS3). We identify novel discourse phenomena that to our knowledge have not been addressed previously (e.g. consistency of verb forms), without requiring _a-priori_ language-specific knowledge. Finally, we design a series of methods to automatically tag words belonging to the identified classes of ambiguities (SS4) and we evaluate existing translation models for different categories of ambiguous translations (SS5).

We examine a parallel corpus spanning 14 language pairs, measuring translation ambiguity and model performance. We find that the context-aware methods, while improving on standard evaluation metrics, only perform significantly better than context-agnostic baselines for certain discourse phenomena in our benchmark. Our benchmark provides a more fine-grained evaluation of translation models and reveals weaknesses of context-aware models, such as verb form cohesion. We also find that DeepL, a commercial document-level translation system, does better in our benchmark than its sentence-level ablation and Google Translate. We hope that the released benchmark and code, as well as our findings, will spur targeted evaluation of discourse phenomena in MT to cover more languages and more phenomena in the future.

## 2 Measuring Context Usage

### Cross-Mutual Information

Past work on contrastive evaluation has examined correct and incorrect translations of specific discourse phenomena [8, 7], but this provides only a limited measure of context usage on phenomena defined by the creators of the dataset. We are therefore interested in devising a metric that is able to capture _all_ context usage by a model, beyond a predefined set.

Conditional Cross-Mutual Information (CXMI) [6, 9] measures the influence of context on model predictions at the corpus level. CXMI is defined as:

\[{\text{CXMI}}( C\to Y|X)=\\ {\text{H}}_{q_{MT_{A}}}(Y|X)-{\text{H}}_{q_{MT_{C}}}(Y|X,C),\]

where \(X\) and \(Y\) are a source and target sentence, respectively, \(C\) is the context, H\({}_{q_{MT_{A}}}\) is the entropy of a _context-agnostic_ MT model, and H\({}_{q_{MT_{C}}}\) refers to a _context-aware_ MT model. This quantity can be estimated over a held-out set with \(N\) sentence pairs and their respective context as:

\[{\text{CXMI}}( C\to Y|X)\approx\\ -\frac{1}{N}\sum_{i=1}^{N}\log\frac{q_{MT_{A}}(y^{(i)}|x^{(i)})}{ q_{MT_{C}}(y^{(i)}|x^{(i)},C^{(i)})}\]

Importantly, the authors find that training a _single_ model \(q_{MT}\) as both the context-agnostic and context-aware model ensures that non-zero CXMI values are due to context and not other factors (see Fernandes et al. (2021) and SS3.1 for details).

Although this approach is promising, it is defined only at a _corpus level_: as the previous equation shows, CXMI is estimated by over a full set of sentences. Since we are interested in measuring how important context is for single sentences or words within a sentence, we extend this definition to capture lower-level context dependency in the next section.

### Context Usage Per Sentence and Word

Pointwise Mutual Information (P-MI) [7] measures the association between two random variables for _specific_ outcomes. Mutual information can be seen as the expected value of P-MI over all possible outcomes of the variables.

Taking inspiration from this, we define the **Pointwise Cross-Mutual Information** (P-CXMI) for a source, target, context triplet \((x,y,C)\) as:

\[{\text{P-CXMI}}( y,x,C)=-\log\frac{q_{MT_{A}}(y|x)}{q_{MT_{C}}(y| x,C)}\]

Intuitively, P-CXMI measures how much more (or less) likely a target sentence \(y\) is when it is given context \(C\), compared to not being given that context. Note that this is estimated _according to the models \(q_{MT_{A}}\) and \(q_{MT_{C}}\)_ since, just like CXMI, this measure depends on their learned distributions.

We can also apply P-CXMI at the _word level_ to measure how much more likely a particular word in a sentence is when it is given the context, by leveraging the auto-regressive property of the neural decoder. Given the triplet \((x,y,C)\) and the word index \(i\), we can measure the P-CXMI for that particular word as:

\[{\text{P-CXMI}}( i,y,x,C)=-\log\frac{q_{MT_{A}}(y_{i}|y_{t<i},x)}{q_{MT_{C}}(y_{i}|y_{ t<i},x,C)}\]

Note that nothing constrains the form of \(C\) or even \(x\) and P-CXMI can, in principle, be applied to any conditional language modelling problem 

We use this metric to find words that are strongly context-dependent, which is to say that their likelihood increases greatly with context relative to other words. These words are the ones that likely correspond to discourse phenomena.

## 3 Which Translation Phenomena Benefit from Context?

To identify salient translation phenomena that require context, we perform a _thematic analysis_Braun and Clarke (2006), examining words with high P-CXMI across different language pairs and manually identifying patterns and categorizing them into phenomena where context is useful for translation.

To do so, we systematically examined (1) the mean P-CXMI per part-of-speech (POS) tag, (2) the words with the highest mean P-CXMI across the corpus, and (3) the individual words with the highest P-CXMI in a particular sentence.

### Data & Model

To compare linguistic phenomena that arise during document-level translation across language pairs, we use a dataset consisting of TED talks' transcripts and translations Qi et al. (2018). We use this dataset due to its abundance of discourse phenomena, as well as its availability across many parallel languages. We study translation between English and Arabic, German, Spanish, French, Hebrew, Italian, Japanese, Korean, Dutch, Portuguese, Romanian, Russian, Turkish and Mandarin Chinese. These 14 target languages are chosen for their high availability of TED talks and linguistic tools, as well as for the diversity of language types in our comparative study (Table 4 in Appendix B). For each language pair, our dataset contains 113,711 parallel training sentences from 1,368 talks, 2,678 development sentences from 41 talks, and 3,385 testing sentences from 43 talks.

To obtain the P-CXMI for words in the data, we train a small Transformer Vaswani et al. (2017) model for every target language and incorporate the target context by concatenating it with the current target sentence Tiedemann and Scherrer (2017). We train the model with _dynamic_ context size Fernandes et al. (2021), by sampling 0-3 target context sentences and estimating P-CXMI by using this model for \(q_{MT_{A}}\) and \(q_{MT_{C}}\) (details in Appendix G).

### Analysis Procedure

We start our analysis by studying POS tags with high mean P-CXMI. In Appendix C, we report the mean P-CXMI for selected POS tags on test data. Some types of ambiguity, such as dual form pronouns (SS3.3), can be linked to a single POS tag and be identified at this step, whereas others require finer inspection.

Next, we inspect the vocabulary items with high mean P-CXMI. At this step, we can detect phenomena that are reflected by certain lexical items that consistently benefit from context for translation.

Finally, we examine individual tokens that obtain the highest P-CXMI. In doing so, we identify patterns that do not depend on lexical features, but rather on syntactic constructions for example. In Table 2, we provide selected examples of tokens that have high P-CXMI and the discourse phenomenon we have identified from them.

### Identified Phenomena

Through our thematic analysis of items with high P-CXMI, we identified various types of translation ambiguity. Unlike previous work, our method requires no prior knowledge of languages and easily

\begin{table}
\begin{tabular}{l|c} \hline \hline _Aville’s mother had HIV virus._**Acville**had the virus, she was born with the virus. & Lexical Cohesion \\ \hline _Your daughter?_**You**nece? & Formality \\ _Write file ?_**VOTE**nece ? & (T-V) \\ \hline _Roger I got’em. Two-Six, this is Two-Six , we’re mobile._ & Formality \\ _I \# IIPLE_2: 2-6 \% 

scales to new languages (SS4.4).

Although this procedure may find phenomena that are intuitive to the annotators, the data-driven approach makes confirmation bias less severe than works relying on introspection. Hence, our procedure can allow us to discover relevant phenomena that have not been previously addressed, such as verb forms. Examples of each phenomenon are given in Table2.

#### 3.3.1 Lexical Cohesion

Entities may have multiple possible translations in the target language, but the same entity should be referred to by the same word in a translated document. This is called lexical cohesion.

#### 3.3.2 Formality

We identify two phenomena which fall under the general category of formality. First, several languages we examined have a T-V distinction (Appendix B, "Pronouns Politeness") in which the second-person pronouns a speaker uses to refer to someone depend on the relationship between the speaker and the addressee.

Second, languages such as Japanese and Korean use honorifics to indicate formality, which are special titles or words expressing courtesy or respect for position.

#### 3.3.3 Pronoun Choice

Unlike in English, many languages use gendered pronouns for pronouns other than the third-person singular, or assign gender based on formal rules rather than semantic ones. In order to assign the correct pronoun, it is therefore necessary to use the previous context to distinguish the grammatical gender of the antecedent.

#### 3.3.4 Verb Form

While English verbs may have five forms 2, other languages may have a more fine-grained verb morphology. For example, English has only a single form for the past tense, while the Spanish past tense consists of six verb forms. Verbs must be translated using the verb form that reflects the tone, mood and cohesion of the document.

Footnote 2: (e.g. _write_, _writes_, _wrote_, _written_, _writing_)

#### 3.3.5 Ellipsis

Ellipsis refers to the omission of superfluous words that are able to be inferred from the context. For instance, in the last row of Table2, the English text does not repeat the verb _know_ in the second sentence as it can be understood from the previous sentence. However, in Turkish, there is no natural way to translate the verb-phrase ellipsis, so context is important for translating the verb correctly.

## 4 Cross-phenomenon MT Evaluation

Next, we we develop a series of methods to automatically tag tokens belonging to these classes of ambiguous translations and propose the **M**ultilingual **D**iscourse-**A**ware (MuDA) benchmark for context-aware MT models.

### MT Evaluation Framework

Given a pair of parallel source and target documents \((X,Y)\), our MuDA tagger assigns one or more tags from a set of discourse phenomena \(\{t^{1}_{i},\cdots,t^{n}_{i}\}\) to each target token \(y_{i}\in Y\). Using the compare-mt toolkit Neubig et al. (2019), we compute the mean word f-measure of system outputs compared to the reference for each tag. This allows us to identify which discourse phenomena models can translate more or less accurately.

### Automatic Tagging

We now describe our taggers for each identified discourse phenomenon. Note that these do **not** require C-XMI to be calculated, and are based on reliable methods for identifying each phenomenon mentioned in subsection3.3. For formality, pronoun choice and verb form, we created language-specific word lists that were verified by native speakers.

Not all phenomena are present in each language. Phenomena that are absent are indicated in Appendix D, as a zero count for that language.

**Lexical Cohesion** To tag words that require lexical cohesion, we first extract

Figure 1: Number of MuDA tags on TED test data. Exact numbers of each tag are given in Appendix D. Number of tags for other document-level datasets can be found in Appendix E.

 

word alignments from a parallel corpus \(D=\{(X_{1},Y_{1}),\cdots,(X_{|D|},Y_{|D|})\}\), where \((X_{m},Y_{m})\) denote the source and target reference document pair. We use the AWESOME aligner Dou2021 to obtain:

\[A_{m}=\{\langle x_{i},y_{j}\rangle~{}|~{}x_{i}\leftrightarrow y_{j},x_{i}\in X _{m},y_{j}\in Y_{m}\},\]

where each \(x_{i}\) and \(y_{j}\) are the lemmatized content source and target words and \(\leftrightarrow\) denotes a bidirectional word alignment. For each target word \(y_{j}\) that is aligned to source word \(x_{i}\), if the alignment pair \(\langle x_{i},y_{j}\rangle\) occurred at least \(3\) times already in the current document, excluding the current sentence, we tag \(y_{j}\) for lexical cohesion 3.

Footnote 3: This threshold of 3 can also be changed within the tagger.

**Formality** For languages with T-V distinction, we tag the target pronouns containing formality distinction if there has previously been a word pertaining to the same formality level in the same document.

Some languages such as Spanish often drop the subject pronoun, and T-V distinction is instead reflected in the verb form. For these languages, we use spaCy HonnibalMontal2017 and Stanza Qi2020 to find POS tags and detect verbs with a second-person subject in the source, and conjugated in the second (T) or third (V) person in the target.

For languages with more complex honorifics systems, such as Japanese, we construct a word list of common honorifics-related words to tag (details in Appendix F.3).

**Pronoun Choice** To find pronouns in English that have multiple translations, we manually construct a list \(P_{\ell}=\{\langle p_{s},\textbf{p}_{t}\rangle\}\) for each language (Appendix F.2), where each \(p_{s}\) is an English pronoun and \(\textbf{p}_{t}\) the list of possible translations of \(p_{s}\) in the language \(\ell\). Then, for each aligned token pair \(\langle x_{i},y_{j}\rangle\), if \(x_{i},y_{j}\) are both pronouns with \(\langle x_{i},\textbf{p}_{t}|y_{j}\in\textbf{p}_{t}\rangle\in P_{\ell}\), and the antecedent of \(x_{i}\) is _not_ in current sentence, we tag \(y_{j}\) as an ambiguous pronoun. To obtain antecedents, we use AllenNLP Gardner2017's coreference resolution module. This procedure is similar to Muller2018.

**Verb Form** For each target language, we define a list \(V_{\ell}=\{v_{1},\cdots,v_{k}\}\) of verb forms (Appendix F.3) where \(v_{i}\in V_{\ell}\) if there exists a verb form in English \(u_{j}\) and an alternate verb form \(v_{k}\neq v_{i}\) in the target language such that an English verb with form \(u_{j}\) may be translated to a target verb with form \(v_{i}\) or \(v_{k}\) depending on the context. Then, for each target token \(y_{j}\), if \(y_{j}\) is a verb of form \(v_{j}\in V_{\ell}\), and another verb with form \(v_{j}\) has appeared previously in the same document, we tag \(y_{j}\) as ambiguous.

**Ellipsis** To detect translation ambiguity due to VP and NP ellipsis, we look for instances where the ellipsis occurs on the source side, but not on the target side, which means that the ellipsis must be resolved during translation. Since existing ellipsis models are limited to specific types of ellipsis, we first train an English (source-side) ellipsis detection model. To do so, we extract an ellipsist dataset from the English data in the Penn Treebank Marcus2019 who trained BERT text classification model Devlin2019, which achieves 0.77 precision and 0.73 recall (see Appendix F.4 for training details). Then, for each sentence pair where the source sentence is predicted to contain an ellipsis, we tag the word \(y_{j}\) in the target sentence \(Y_{m}\) if: (1) \(y_{j}\) is a verb, noun, proper noun or pronoun; (2) \(y_{j}\) has occurred in the previous target sentences of the same document; (3) \(y_{j}\) is not aligned to any source words, that is, \(\not\exists\,x_{i}\in X_{m}\) s.t. \(\langle x_{i},y_{j}\rangle\in A_{m}\).

### Evaluation of Automatic Tags

We apply the MuDA tagger to the reference translations of our TED talk data. We thus obtain an evaluation set of 3,385 parallel sentences for each of the 14 language pairs. In Appendix C we report the mean P-CXMI for each language and MuDA tag. Overall, we find higher P-CXMI on tokens with a tag compared to those without, which provides empirical evidence that models indeed rely on context to predict words with MuDA tags.

Appendix D shows that the frequency of tags varies significantly across languages. Overall, only 4.5% of the English sentences have been marked for ellipsis, giving an upper bound for the number of ellipsis tags in other languages. We find that languages from a different family than English have a relatively high number of ellipsis tags. We also find that Korean and especially Japanese have more formality tags than languages with T-V distinction, which reflects that register is more often important when translating to languages with honorifics.

**Manual Evaluation** To evaluate our tagger, we asked native speakers with computational linguistics backgrounds to manually verify MuDA tags for 8 languages on 50 randomly selected utterances as well as all words tagged with _ellipisis_ in our corpus. This allows us to measure how many automatic 

tags violate the given definition of the linguistic tag. Table 3 reports the tags' precision 4.

Footnote 4: Workers were paid 205/hour.

For all languages, we obtain high precision for all tags except _ellipsis_, confirming that the methodology can scale to languages where no native speakers were involved in developing the tags. For _ellipsis_, false positives often come from one-to-many or non-literal translations, where the aligner does not align all target words to the corresponding source word. We believe that the _ellipsis_ tagger is still useful in selecting difficult examples that require context for translation; despite the low precision, we find a significantly higher P-CXMI on _ellipsis_ words for many languages (Appendix C).5

Footnote 5: Also note that wrongly assigned tags should also not penalize a system greatly as it should give a low score only if the translation does not match the falsely tagged word.

### Extension to New Languages

While MuDA currently supports 14 language pairs, our methodology can be easily extended to new languages. The _lexical_ and _ellipsis_ tags can be directly applied to other languages provided a word aligner between English and the new target language. The _formality_ tag can be extended by adding a list of pronouns or verb forms related to formality in the new language. Similarly, the _pronouns_ and _verb forms_ tag can also be extended by providing a list of ambiguous pronouns and verb forms.

Exhaustively listing all relevant phenomena in document-level MT is extremely complex and beyond the scope of our paper. To identify new discourse phenomena on other languages, our thematic analysis can be reused as follows: (1) Train a model with dynamic context size on translation between the new language pair; (2) Use the model to compute P-CXMI for words in a parallel document-level corpus of the language pair; (3) Manually analyze the POS tags, vocabulary items and individual tokens with high P-CXMI; (4) Link patterns of tokens with high P-CXMI to particular discourse phenomena by consulting linguistic resources.

## 5 Exploring Context-aware MT

Our MuDA trigger can be applied to documents in the supported languages to create benchmarking datasets for discourse phenomena during translation. We use our benchmark of the TED talk dataset enhanced with MuDA tags to perform an exploration of context usage across languages with 4 models, including commercial systems.

### Trained Models

We train a sentence-level and document-level concatenation-based small transformer (base) for every target language. While conceptually simple, concatenation approaches have been shown to outperform more complex models when properly trained. For the context-aware model, the major difference from SS3.1 is that we use a _static_ context size of 3, since we are not using these models to measure P-CXMI. Lopes et al. (2020).

To evaluate stronger models, we additionally train a large transformer model (large) that was pretrained on a large, sentence-level corpora, for German, French, Japanese and Chinese. Further details can be found in Appendix G.

### Commercial Models

To assess if commercially available machine translation engines are able to leverage context and therefore do well in MuDA, we consider two engines:6 (1) the _Google Cloud Translation_ v2 API. In early experiments, we assessed that this model only does sentence-level translation, but included it due to its widespread usage; (2) the _DeepL_ v2 API. This model advertises its usage of context as part of translations and our experiments confirm this. Early experimentation with other providers Amazon and Azure) indicated that these are not context-aware so we refrained from evaluating them.

Footnote 6: translate.google.com, deepl.com. Translations were obtain from version of engines available in April 2021

To obtain provider translations, we feed the documents into an API request. To re-segment the translation into sentences, we include special marker tokens in the source that are preserved during translation and split the translation on those tokens. We also evaluate a _sentence-level_ version of DeepL where we feed each sentence separately to compare with its docu

\begin{table}
\begin{tabular}{c|r r r r r} \hline  & lexical & formality & pronouns & verb form & ellipsis \\ \hline es & 1.00 & 0.92 & 1.00 & 1.00 & 0.53 \\ fr & 1.00 & 1.00 & 1.00 & 0.94 & 0.43 \\ ja & 1.00 & 1.00 & 1.00 & – & 0.41 \\ ko & 1.00 & 0.94 & – & – & 0.26 \\ pt & 0.99 & 0.88 & 1.00 & – & 0.31 \\ ru & 1.00 & 1.00 & – & 1.00 & 0.50 \\ tr & 1.00 & 1.00 & – & 1.00 & 0.57 \\ zh & 1.00 & 1.00 & – & – & 0.78 \\ \hline \end{tabular}
\end{table}
Table 3: Precision of MuDA tags on 50 utterances.

 



### Results and Discussion

Figure 2 shows results for base models, trained either without (no-context) or with context, and for the latter with either _predicted_ (context) or _reference_ context (context-gold) during decoding. Results are reported with respect to standard MT metrics BLEU Papineni et al. (2002) and COMET Rei et al. (2020), as well as the MuDA benchmark. The corpus-level metrics BLEU and COMET are calculated over the entire corpus, rather than just the sentences tagged by MuDA.

First, we find that BLEU scores are highest for context-gold models for most language pairs, but context-agnostic models have higher COMET scores. Moreover, in terms of mean word f-measure overall, we do not find significant differences between the three systems. It is therefore difficult to see which system performs the best on document-level ambiguities using only corpus-level metrics.

For words tagged by MuDA as requiring context for translation, context-aware models often achieve significantly higher word f-measure than context-agnostic models on certain tags such as _ellipsis_ and _formality_, but not on other tags such as _lexical_ and _verb form_. This demonstrates how MuDA allows us to clarify which inter-sentential ambiguities context-aware models are able to resolve.

For the pretrained large models (Figure 3), context-aware models perform better than the context-agnostic on corpus-level metrics, especially COMET. On words tagged with MuDA, context-aware models generally obtain the high

Figure 3: Impact of context on BLEU, COMET, and Word f-measure per tag for large models. Values that are statistically significantly higher than no-context (\(p<0.05\)) are marked with *. Languages for which the phenomenon doesn’t exist are marked with \(\emptyset\). BLEU scores are normalized between [0,1]

Figure 2: Impact of context on BLEU, COMET, and Word f-measure per tag for base context-aware models. BLEU, COMET and word f-measures statistically significantly higher than no-context (\(p<0.05\)) are marked with *. Languages for which the phenomenon doesn’t exist are marked with \(\emptyset\). BLEU scores are normalized between [0,1]

est f-measure as well, particularly when given reference context, especially on phenomena such as _lexical_ and _pronouns_, but improvements are less pronounced than on corpus-level evaluation.

Among commercial engines (Figure 4), DeepL outperforms Google on most metrics and language pairs. The sentence-level ablation of DeepL performs worse than its document-level system for most MuDA tags.

Current context-aware MT systems translate some inter-sentential discourse phenomena well, but are unable to consistently obtain significant improvements over context-agnostic counterparts on challenging MuDA data. Tables with all results can be found in Appendix H.

## 6 Related Work

Several works have worked on measuring the performance of MT models on contextual discourse phenomena. The first example of this was done by Hardmeier et al. (2010), which evaluated automatically the precision and recall of pronoun translation in statistical MT systems. Jwalapuram et al. (2019) proposed evaluating models on pronoun translation based on a pairwise comparison between translations that were generated with and without context, and later Jwalapuram et al. (2020) extended this work to include more languages and phenomena in their automatic evaluation/test set creation. These works rely on prior domain knowledge and intuition to identify context-aware phenomena, whereas we take a systematic, data-driven approach.

Most works have focused on evaluating performance in discourse phenomena through the use of _contrastive datasets_. Muller et al. (2018) automatically create a dataset for anaphoric pronoun resolution to evaluate MT models in EN \(\rightarrow\) DE. Bawden et al. (2018) manually creates a dataset for both pronoun resolution and lexical choice in EN \(\rightarrow\) FR. Voita et al. (2018, 2019b) creates a dataset for anaphora resolution, deixis, ellipsis and lexical cohesion in EN \(\rightarrow\) RU. However, Yin et al. (2021) suggest that _translating_ and _disambiguating_ between two contrastive choices are inherently different, motivating our approach in measuring direct translation performance.

## 7 Conclusions and Future Work

We investigate types of ambiguous translations where MT models benefit from context using our proposed P-CXMI metric. We perform a data-driven thematic analysis across 14 languages to identify context-sensitive discourse phenomena,

Figure 4: Scores for commercial models. DeepL (doc) BLEU, COMET and word f-measures statistically significantly higher than DeepL (sent) are marked with *. Languages for which neither DeepL or Google translations are available are marked with \(\emptyset\). BLEU scores are normalized between [0,1]



some of which (such as _verb forms_) have not been previously addressed in work on MT. In comparison to previous work, our approach is systematic, extensible, and does not require prior knowledge of the language. Additionally, the P-CXMI metric can be used to identify other context-dependent words in generation. We construct the MuDA benchmark that tags words in parallel corpora and evaluates models on 5 context-dependent phenomena. Our evaluation reveals that context-aware and commercial translation systems achieve small improvements over context-agnostic models on our benchmark, and we encourage further development of models that improve on context-aware translation.

### Limitations

While MuDA relies on set of hand-crafted rules for tagging specific phenomena, these rules might involve the use of other error-prone systems (such as coreference resolution and alignment models) and these errors might be susceptible to problems (such as lack of out-of-domain generalization) that could limit the applicability of our tagger. However, this could be fixed by extending MuDA to use newer and better versions of these systems.

The use of F-1 per tag with surface-form matching between reference/translation can also lead to penalizing translations that use context correctly but choose other equivalent words. Nevertheless, this should also be mitigable by extending the scoring method to, for example, match synonyms.

Finally, the benchmarking of context-aware models might not apply to newer, state-of-the-art translation models, especially if these leverage large language models that were trained on long-context data.

## Acknowledgements

We would like to thank Uri Alon, Ipek Baris, George Bejinariu, Hiba Belkadi, Chloe Billiotte, Giovanni Campagna, Remi Castera, Volkan Cirik, Taisiya Glushkova, Junxian He, Mert Inan, Alina Karakanta, Benno Krojer, Emma Landry, Chayoung Park, Artidoro Pagnoni, Maria Ryskina, Odette Scharenborg, Melanie Sclar, Jenny Seok, Emma Schippers, Bogdan Vasilescu for advice on various languages and help with manual annotations.

We would also like to thank all the members of DeepSPIN and NeuLab who provided feedback on earlier versions of this work. This work was supported by the European Research Council (ERC StG DeepSPIN 758969), by EU's Horizon Europe Research and Innovation Actions (UTTER, contract 101070631), by the P2020 program MAIA (LISBOA-01-0247-FEDER-045909), by the Portuguese Recovery and Resilience Plan through project C645008882-00000055 (NextGenAI, Center for Responsible AI), and by the Fundacao para a Ciencia e Tecnologia through contracts SFRH/BD/150706/2020 and UIDB/50008/2020.

## References

* D. Berrault, A. Bird, and B. Haddow (2018)Evaluating discourse phenomena in neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1304-1313. External Links: Link Cited by: SS1.
* E. Bugliarello, S. J. Mielke, A. Anastasopoulos, R. Coterell, and N. Okazaki (2020)It's easier to translate out of English than into it: measuring neural translation difficulty by cross-mutual information. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 1640-1649. External Links: Link Cited by: SS1.
* M. Cettolo, C. Girardi, and M. Federico (2012)WIT3: web inventory of transcribed and translated talks. In Proceedings of the 16th Annual conference of the European Association for Machine Translation, Trento, Italy, pp. 261-268. External Links: Link Cited by: SS1.
* K. W. Church and P. Hanks (1990)Word association norms, mutual information, and lexicography. Computational Linguistics16 (1), pp. 22-29.

 

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics.
* Dou and Neubig [2021] Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online. Association for Computational Linguistics.
* Espla et al. [2019] Miquel Espla, Mikel Forcada, Gema Ramirez-Sanchez, and Hieu Hoang. 2019. ParaCrawl: Web-scale parallel corpora for the languages of the EU. In Proceedings of Machine Translation Summit XVII Volume 2: Translator, Project and User Tracks, pp. 118-119, Dublin, Ireland. European Association for Machine Translation.
* Fernandes et al. [2021] Patrick Fernandes, Kayo Yin, Graham Neubig, and Andre F. T. Martins. 2021. Measuring and increasing context usage in context-aware machine translation. In Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP), Virtual.
* Gardner et al. [2017] Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. Allennlp: A deep semantic natural language processing platform.
* Guillou et al. [2018] Liane Guillou, Christian Hardmeier, Ekaterina Lapshinova-Koltunski, and Sharid Loiciga. 2018. A pronoun test suite evaluation of the English-German MT systems at WMT 2018. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, Belgium, Brussels. Association for Computational Linguistics.
* Hardmeier et al. [2010] Christian Hardmeier, Marcello Fondazione, and Bruno Kessler. 2010. Modelling pronominal anaphora in statistical machine translation.
* Honnibal and Montani [2017] Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
* Jwalapuram et al. [2019] Prathyusa Jwalapuram, Shafiq Joty, Irina Temnikova, and Preslav Nakov. 2019. Evaluating promominal anaphora in machine translation: An evaluation measure and a test suite. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China. Association for Computational Linguistics.
* Jwalapuram et al. [2020] Prathyusa Jwalapuram, Barbara Rychalska, Shafiq R. Joty, and Dominika Basaj. 2020. Can your context-aware MT system pass the dip benchmark tests? : Evaluation benchmarks for discourse phenomena in machine translation. CoRRabs/2004.14607.
* Laubli et al. [2018] Samuel Laubli, Rico Senrrich, and Martin Volk. 2018. Has machine translation achieved human parity? a case for document-level evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4791-4796, Brussels, Belgium. Association for Computational Linguistics.
* Lopes et al. [2020] Antonio Lopes, M. Amin Farajian, Rachel Bawden, Michael Zhang, and Andre F. T. Martins. 2020. Document-level neural MT: A systematic comparison. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 225-234, Lisboa, Portugal. European Association for Machine Translation.
* Marcus et al. [1993] Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. _Computational Linguistics_, 19(2):313-330.
* Maruf and Haffari [2018] Sameen Maruf and Gholamreza Haffari. 2018. Document context neural machine translation with memory networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia. Association for Computational Linguistics.
* Maruf et al. [2021] Sameen Maruf, Fahimeh Saleh, and Gholamreza Haffari. 2021. A survey on document-level neural machine translation: Methods and evaluation. ACM Comput. Surv., 54(2).
* Miculicich et al. [2018] Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018. Document-level neural machine translation with hierarchical attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2947-2954, Brussels, Belgium. Association for Computational Linguistics.
* Morishita et al. [2020] Makoto Morishita, Jun Suzuki, and Masaaki Nagata. 2020. JParaCrawl: A large scale web-based English-Japanese parallel corpus. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 3603-3609, Marseille, France. European Language Resources Association.
* Muller et al. [2018] Mathias Muller, Annette Rios, Elena Voita, and Rico Sennrich. 2018. A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, Brussels, Belgium. Association for Computational Linguistics.
* Neubig et al. [2019] Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthil, and Xinyi Wang. 2019. compare-mt: A tool for holistic comparison of language generation systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), Minneapolis, Minnesota. Association for Computational Linguistics.

 

* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angel Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. Fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 101-108, Online. Association for Computational Linguistics.
* Qi et al. (2018) Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 529-535, New Orleans, Louisiana. Association for Computational Linguistics.
* Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2685-2702, Online. Association for Computational Linguistics.
* Tiedemann and Scherrer (2017) Jorg Tiedemann and Yves Scherrer. 2017. Neural machine translation with extended context. In _Proceedings of the Third Workshop on Discourse in Machine Translation_, pages 82-92, Copenhagen, Denmark. Association for Computational Linguistics.
* Toral et al. (2018) Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the unattainable? reassessing claims of human parity in neural machine translation. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 113-123, Brussels, Belgium. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 5998-6008.
* Voita et al. (2019a) Elena Voita, Rico Sennrich, and Ivan Titov. 2019a. Context-aware monolingual repair for neural machine translation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 877-886, Hong Kong, China. Association for Computational Linguistics.
* Voita et al. (2019b) Elena Voita, Rico Sennrich, and Ivan Titov. 2019b. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsik, and lexical cohesion. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1198-1212, Florence, Italy. Association for Computational Linguistics.
* Voita et al. (2018) Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine translation learns anaphora resolution. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1264-1274, Melbourne, Australia. Association for Computational Linguistics.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Phu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Yin et al. (2021) Kayo Yin, Patrick Fernandes, Danish Pruthi, Aditi Chaudhary, Andre F. T. Martins, and Graham Neubig. 2021. Do context-aware translation models pay the right attention? In _Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP)_, Virtual.

 



## Appendix A MuDA Toolkit Usage

To tag an existing dataset and extract the tags for later use, run the following command:

```
1pythonmuda/main.py \
2--src/path/to/src \
3--tgt/path/to/tgt \
4--docids/path/to/docids \
5--dump-tags/tmp/maia_ende.tags \
6--tgt-langlang
```

To evaluate models on a particular dataset (reporting per-tag metrics dicussed in this paper), run the following command:

```
1pythonmuda/main.py \
2--src/path/to/src \
3--tgt/path/to/tgt \
4--docids/path/to/decids \
5--hyps/path/to/hyps.ml/path/to/hyps.m2 \
6--tgt-langlang
```

## Appendix B Language Properties

Table 4 summarizes the properties of the languages analyzed in this work.

## Appendix C P-CXMI Results

Table 5 presents the average P-CXMI value per POS tag and per MuDA tag.

### Tag Numbers

Table 6 lists the counts of each tag per language.

## Appendix E Tagging other Document-level Datasets

We report the number of tags found for two other document-level datasets commonly used in the literature: (1) IWSLT-17 Cettolo et al. (2012) test sets for EN \(\rightarrow\) DE and EN \(\rightarrow\) FR and (2) A randomly subsampled portion of the news-commentary dataset for EN \(\rightarrow\{\text{AR},\text{DE},\text{ES},\text{FR},\text{NL},\text{PT}, \text{RU},\text{ZH}\}\)Barrault et al. (2019). These results can be found respectively in Figure 5 and Figure 6.

\begin{table}
\begin{tabular}{c|c c c c c} \hline \hline Language & Family & Word Order & Pronouns Politeness & Gendered Pronouns & Gender Assignment \\ \hline Arabic & Afro-Asiatic & VSO & None & 1 and/or 2 and 3 & Semantic-Formal \\ English & Indo-European & SVO & None & 3.Sing & Semantic \\ German & Indo-European & SOV/SVO & Binary & 3.Sing & Semantic-Formal \\ Spanish & Indo-European & SVO & Binary & 1 and/or 2 and 3 & Semantic-Formal \\ French & Indo-European & SVO & Binary & 3.Sing & Semantic-Formal \\ Hebrew & Afro-Asiatic & SVO & None & 1 and/or 2 and 3 & Semantic-Formal \\ Italian & Indo-European & SVO & Binary & 3.Sing & Semantic-Formal \\ Japanese & Japonic & SOV & Avoided & 3 & None \\ Korean & Koreanic & SOV & Avoided & 3.Sing & None \\ Dutch & Indo-European & SOV/SVO & Binary & 3.Sing & Semantic-Formal \\ Portuguese & Indo-European & SVO & Binary & 3.Sing & Semantic-Formal \\ Romanian & Indo-European & SVO & Multiple & 3.Sing & Semantic-Formal \\ Russian & Indo-European & SVO & Binary & 3.Sing & Semantic-Formal \\ Turkish & Turkic & SOV & Binary & None & None \\ Mandarin & Sino-Tibetan & SVO & Binary & 3.Sing & None \\ \hline \hline \end{tabular}
\end{table}
Table 4: Properties of the languages in our study.



Figure 5: Number of tags for EN \(\rightarrow\) DE and EN \(\rightarrow\) FR in the IWSLT17 dataset. Lexical cohesion and verb form are common phenomena in this dataset.

Figure 6: Number of tags across languages in the news-commentary dataset. Lexical cohesion and verb form are common phenomena in this dataset.

 



## Appendix F Tagger Details

### Formality Words

Table 7 gives the list of words related to formality for each target language.

### Ambiguous Pronouns

Table 8 provides English pronouns and the list of possible target pronouns.

### Ambiguous Verbs

Table 9 lists verb forms that may require disambiguation during translation.

### Ellipsis Classifier

We train a BERT text classification model Devlin et al. (2019) on data from the Penn Treebank, where we labeled each sentence containing the tag '*??" as containing ellipsis Bies et al. (1995). We obtain 248,596 sentences total, with 2,863 tagged as ellipsis. Then, our model using HuggingFace Transformers Wolf



et al., 2019). To address the imbalance in labels, we up-weight the loss for samples tagged as ellipsis by a factor of 100.

## Appendix G Training details

The _transformer-small_ model has hidden size of 512, feedforward size of 1024, 6 layersa and 8 attention heads. The _transformer-large_ model has hidden size of 1024, feedforward size of 4096, 6 layers, 16 attention heads.

As in Vaswani et al. (2017), we train using the Adam optimizer with \(\beta_{1}=0.9\) and \(\beta_{2}=0.98\) and use an inverse square root learning rate scheduler, with an initial value of \(10^{-4}\) for large model and \(5\times 10^{-4}\) for the base and multi models, with a linear warm-up in the first \(4000\) steps.

For the pretrained models we used Paracrawl (Espla et al., 2019) for German and French, JParacrawl (Morishita et al., 2020) for Japanese and the Backtranslated News from WMT2021 for Chinese.

Due to the sheer number of experiments, we use a single seed per experiment.

We base our experiments on the framework _Fairseq_(Ott et al., 2019).
 



\begin{table}
\begin{tabular}{c|r c} de & du \\  & sie \\ \hline es & tu, tu, tus, ti, contigo, tuyo, te, tuya \\  & used, vosotros, vuestro, vuestra, vuestras, os \\ \hline fr & tu, ton,ta, tes, toi, te, tien, tiens, tienne, tiennes \\  & vous, votere, vs \\ \hline it & tu, tuo, tuna, tuoi \\  & lei, suo, sua, suoi \\ \hline ja & \(\vec{\iota}_{2}\), \(\vec{\iota}_{3}\), \(\vec{\iota}_{4}\), \(\vec{\iota}_{5}\), \(\vec{\iota}_{6}\), \(\vec{\iota}_{7}\), \(\vec{\iota}_{8}\), \(\vec{\iota}_{9}\), \(\vec{\iota}_{10}\), \(\vec{\iota}_{11}\), \(\vec{\iota}_{12}\), \(\vec{\iota}_{13}\), \(\vec{\iota}_{14}\), \(\vec{\iota}_{15}\), \(\vec{\iota}_{16}\), \(\vec{\iota}_{17}\), \(\vec{\iota}_{18}\), \(\vec{\iota}_{19}\), \(\vec{\iota}_{19}\), \(\vec{\iota}_{12}\), \(\vec{\iota}_{13}\), \(\vec{\iota}_{14}\), \(\vec{\iota}_{15}\), \(\vec{\iota}_{16}\), \(\vec{\iota}_{17}\), \(\vec{\iota}_{18}\), \(\vec{\iota}_



\begin{table}
\begin{tabular}{c c|c} \hline ar & you & _Last! i, juli



\begin{table}
\begin{tabular}{c c|r r r r r r r r r r r r r} \hline  & & ar & de & es & fr & he & it & ja & ko & nl & pt & ro & ru & tr & zh \\ \hline  & no-context & 17.25 & 28.02 & 35.72 & 37.74 & 32.70 & 32.30 & 7.10 & 6.80 & 32.22 & 39.03 & 25.36 & 17.00 & 12.32 & 15.96 \\ BLEU & context & 16.92 & 28.24 & 36.00 & 37.23 & 32.92 & 32.11 & 4.48 & 3.77 & 32.67 & 39.10 & 25.37 & 17.14 & 11.97 & 15.01 \\  & context-gold & 18.61 & 28.60 & 36.27 & 37.96 & 33.41 & 32.37 & 5.96 & 6.92 & 32.73 & 39.55 & 28.49 & 17.70 & 12.49 & 16.05 \\ \hline  & no-context & 0.002 & 0.1841 & 0.3809 & 0.3087 & 0.0498 & 0.2608 & -0.5366 & 0.0275 & 0.3105 & 0.4562 & 0.3826 & 0.0033 & 2.113 & -0.1419 \\ COMET & context & -0.0066 & 0.1846 & 0.3875 & 0.2181 & 0.0887 & 0.2496 & -0.7728 & -0.3393 & 0.3238 & 0.4444 & 0.3474 & 0.0190 & 10.831 & -0.1917 \\  & context-gold & 0.0025 & 0.1886 & 0.3879 & 0.2821 & 0.0922 & 0.2467 & -0.6827 & -0.1000 & 0.3218 & 0.4506 & 0.3805 & -0.0173 & 0.1871 & -0.1274 \\ \hline \hline  & no-context & 0.374 & 0.387 & 0.210 & 0.400 & 0.439 & 0.259 & 0.123 & 0.169 & 0.400 & 0.342 & 0.333 & 0.255 & 0.165 & 0.145 \\ ellipsis & contract & 0.325 & 0.323 & 0.333 & 0.406 & 0.389 & 0.400 & 0.021 & 0.033 & 0.471 & 0.450 & 0.270 & 0.292 & 0.240 & 0.135 \\  & context-gold & 0.388 & 0.296 & 0.300 & 0.435 & 0.371 & 0.381 & 0.025 & 0.150 & 0.444 & 0.450 & 0.306 & 0.226 & 0.187 & 0.154 \\ \hline  & no-context & – & 0.607 & 0.370 & 0.792 & – & 0.429 & 0.443 & 0.399 & 0.682 & 0.599 & 0.434 & 0.464 & 0.097 & 0.691 \\ formality & context & – & 0.639 & 0.351 & 0.791 & – & 0.462 & 0.414 & 0.397 & 0.694 & 0.600 & 0.405 & 0.469 & 0.083 & 0.695 \\  & context-gold & – & 0.661 & 0.443 & 0.803 & – & 0.464 & 0.431 & 0.425 & 0.697 & 0.622 & 0.440 & 0.492 & 0.182 & 0.741 \\ \hline  & no-context & 0.639 & 0.762 & 0.819 & 0.826 & 0.723 & 0.766 & 0.615 & 0.574 & 0.821 & 0.853 & 0.661 & 0.624 & 0.671 & 0.645 \\ lexical & context & 0.630 & 0.736 & 0.833 & 0.830 & 0.722 & 0.772 & 0.572 & 0.524 & 0.825 & 0.851 & 0.689 & 0.624 & 0.647 & 0.644 \\  & context-gold & 0.675 & 0.737 & 0.832 & 0.832 & 0.727 & 0.773 & 0.614 & 0.593 & 0.828 & 0.857 & 0.713 & 0.625 & 0.647 & 0.676 \\ \hline  & no-context & 0.660 & 0.613 & 0.576 & 0.774 & – & 0.548 & 0.473 & – & – & 0.452 & 0.356 & – & – & – \\ pronouns & context & 0.691 & 0.614 & 0.538 & 0.771 & – & 0.549 & 0.377 & – & – & 0.451 & 0.414 & – & – & – \\  & contact-gold & 0.700 & 0.624 & 0.550 & 0.788 & – & 0.530 & 0.428 & – & – & 0.485 & 0.432 & – & – & – \\ \hline  & no-context & – & – & 0.263 & 0.435 & 0.227 & 0.308 & – & – & 0.477 & – & 0.292 & 0.215 & 0.128 & – \\ verb tense & context & – & – & 0.287 & 0.442 & 0.229 & 0.282 & – & – & 0.479 & – & 0.292 & 0.215 & 0.094 & – \\  & context-gold & – & – & 0.272 & 0.435 & 0.229 & 0.285 & – & – & 0.487 & – & 0.328 & 0.238 & 0.120 & – \\ \hline \hline \end{tabular}
\end{table}
Table 9: Ambiguous verb forms w.r.t. English for each target language.





\begin{table}
\begin{tabular}{c c|r r r r r r} \hline \hline  & & de & fr & ja & zh \\ \hline \multirow{4}{*}{BLEU} & no-context & 36.09 & 45.64 & 15.55 & 22.15 \\  & context & 35.86 & 45.40 & 12.68 & 22.68 \\  & context-gold & 36.69 & 46.60 & 16.60 & 22.98 \\ \hline \multirow{4}{*}{COMET} & no-context & 0.5256 & 0.6332 & 0.0602 & 0.1160 \\  & context & 0.5337 & 0.6425 & 0.0753 & 0.2705 \\  & context-gold & 0.5427 & 0.6529 & 0.1808 & 0.2809 \\ \hline \hline \multirow{4}{*}{ellipsis} & no-context & 0.429 & 0.462 & 0.126 & 0.254 \\  & context & 0.518 & 0.393 & 0.068 & 0.230 \\  & context-gold & 0.444 & 0.444 & 0.144 & 0.209 \\ \hline \multirow{4}{*}{formality} & no-context & 0.642 & 0.824 & 0.510 & 0.747 \\  & context & 0.640 & 0.810 & 0.513 & 0.739 \\  & context-gold & 0.692 & 0.820 & 0.537 & 0.739 \\ \hline \multirow{4}{*}{lexical} & no-context & 0.773 & 0.864 & 0.704 & 0.661 \\  & context & 0.776 & 0.868 & 0.699 & 0.671 \\  & context-gold & 0.796 & 0.875 & 0.740 & 0.696 \\ \hline \multirow{4}{*}{pronouns} & no-context & 0.633 & 0.790 & 0.493 & – \\  & context & 0.635 & 0.795 & 0.541 & – \\  & context-gold & 0.665 & 0.801 & 0.536 & – \\ \hline \multirow{4}{*}{verb tense} & no-context & – & 0.526 & – & – \\  & context & – & 0.532 & – & – \\  & context-gold & – & 0.534 & – & – \\ \hline \hline \end{tabular}
\end{table}
Table 11: Word f-measure per tag for large models. BLEU, COMET, word f-measures statistically significantly higher than no-context (_p_ ¡ 0.05) are underlined.

\begin{table}
\begin{tabular}{l l|r r r r r r r r r r r r r} \hline \hline  & & ar & de & es & fr & he & it & ja & ho & nl & pt & ro & ru & ur & zh \\ \hline \multirow{4}{*}{BLEU} & Google & 11.73 & 34.76 & 43.47 & 30.77 & 10.77 & 31.34 & 12.98 & 8.77 & 38.51 & 38.49 & 28.54 & 24.79 & 18.22 & 28.92 \\  & DeepL (sent) & x & 34.29 & 42.00 & 42.57 & x & 35.41 & 14.88 & s & 37.58 & 37.37 & 28.98 & 25.67 & x & 27.94 \\  & DeepL (doc) & x & 36.75 & 43.06 & 43.43 & x & 36.04 & 15.66 & x & 38.29 & 37.76 & 29.79 & 26.53 & x & 27.34 \\ \hline \multirow{4}{*}{COMET} & Google & 0.3862 & 0.5480 & 0.7694 & 0.6655 & 0.3666 & 0.6707 & 0.2116 & 0.4721 & 0.6401 & 0.7925 & 0.7437 & 0.5121 & 0.7254 & 0.3697 \\  & DeepL (sent) & x & 0.5750 & 0.7680 & 0.7121 & x & 0.6951 & 0.2973 & x & 0.6321 & 0.7513 & 0.8026 & 0.5501 & x & 0.3739 \\  & DeepL (doc) & x & 0.5848 & 0.7882 & 0.7267 & x & 0.7049 & 0.2343 & x & 0.6357 & 0.7572 & 0.8121 & 0.5495 & x & 0.2453 \\ \hline \hline \multirow{4}{*}{ellipsis} & Google & 0.343 & 0.667 & 0.500 & 3.306 & 0.359 & 0.468 & 0.279 & 0.352 & 0.389 & 0.632 & 0.405 & 0.367 & 0.236 & 0.323 \\  & DeepL (sent) & x & 0.417 & 0.400 & 4.422 & x & 0.500 & 0.275 & x & 0.500 & 0.421 & 0.458 & 0.385 & x & 0.303 \\  & DeepL (doc) & x & 0.435 & 0.526 & 0.493 & x & 0.553 & 0.208 & x & 0.500 & 0.359 & 0.532 & 0.385 & x & 0.295 \\ \hline \multirow{4}{*}{formality} & Google & – & 0.621 & 0.404 & 0.738 & – & 0.458 & 0.489 & 0.300 & 0.638 & 0.633 & 0.479 & 0.512 & 0.367 & 0.599 \\  & DeepL (sent) & – & 0.641 & 0.419 & 0.733 & – & 0.455 & 0.487 & x & 0.610 & 0.625 & 0.533 & 0.533 & x & 0.729 \\  & DeepL (doc) & – & 0.670 & 0.446 & 0.785 & – & 0.503 & 0.520 & x & 0.641 & 0.614 & 0.526 & 0.534 & x & 0.664 \\ \hline \multirow{4}{*}{lexical} & Google & 0.665 & 0.786 & 0.854 & 0.827 & 0.697 & 0.794 & 0.602 & 0.611 & 0.825 & 0.860 & 0.700 & 0.635 & 0.677 & 0.693 \\  & DeepL (sent) & x & 0.773 & 0.840 & 0.860 & x & 0.805 & 0.657 & x & 0.799 & 0.848 & 0.714 & 0.653 & x & 0.660 \\  & DeepL (doc) & x & 0.776 & 0.841 & 0.872 & x & 0.812 & 0.640 & x & 0.802 & 0.846 & 0.713 & 0.649 & x & 0.657 \\ \hline \multirow{4}{*}{pronouns} & Google & 0.670 & 0.648 & 0.626 & 0.757 & – & 0.511 & 0.486 & – & — & 0.488 & 0.326 & – & – & – \\  & DeepL (sinc) & x & 0.608 & 0.538 & 0.737 & – & 0.543 & 0.526 & – & 0.483 & 0.394 & – & – & – \\  & DeepL (doc) & x & 0.706 & 0.558 & 0.789 & – & 0.551 & 0.557 & – & 0.513 & 0.472 & – & – & – \\ \hline \multirow{4}{*}{verb tense} & Google & – & – & 0.415 & 0.529 & 0.311 & 0.450 & – & – & 0.554 & – & 0.358 & 0.314 & 0.167 & – \\  & DeepL (sent) & – & – & 0.390 & 0.553 & x & 0.478 & – & – & 0.562 & – & 0.400 & 0.327 & x & – \\ \cline{1-1}  & DeepL (doc) & – & – & 0.426 & 0.562 & x & 0.445 & – & – & 0.567 & – & 0.411 & 0.349 & x & – \\ \hline \hline \end{tabular}
\end{table}
Table 12: Scores for commercial models. DeepL (doc) BLEU, COMET and word f-measures statistically significantly higher than DeepL (sent) are underlined.





**ACL 2023 Responsible NLP Checklist**

A For every submission: A1. Did you describe the limitations of your work? _Last section in page 9 (unnumber) A2. Did you discuss any potential risks of your work? Work doesn't have immediate ethical risk_ A3. Do the abstract and introduction summarize the paper's main claims? _Section 1 and Abstract_ A4. Have you used AI writing assistants when working on this paper? _Left blank. B Did you use or create scientific artifacts? Section 4 and 5 B1. Did you cite the creators of artifacts you used? Section 4 and 5_ B2. Did you discuss the license or terms for use and / or distribution of any artifacts? _Datasets used are commonly used by the community, and the (permissive) license for our tagger is in the official code repository_ B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? _Datasets used are commonly used by the community_ B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? _Not applicable. Left blank._ B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? _Section 4_ B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. _Section 4_ C Did you run computational experiments? _Section 5_ C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? _Section 5.1 and Appendix F

C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? _Section 5.1 and Appendix F_ C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? _Section 5_ C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? _Section 4 and 5_ D **D****id you use human annotators (e.g., crowdworkers) or research with human participants?**
* [leftmargin=0cm]
* D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? _We provide a brief description (but not full text instructions) in section 4.3_ D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants' demographic (e.g., country of residence)? _Section 4.3_ D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? _Not applicable. Left blank._ D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? _Not applicable. Left blank._ D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? _Section 4.3_