DiP Benchmark Tests: Evaluation Benchmarks for Discourse Phenomena in MT
Authors: Prathyusha Jwalapuram
Barbara Rychalska
Shafiq Joty
Dominika Basaj
Venue:  (2021)
Keywords: machine translation
discourse
evaluation
benchmark
testsets
leaderboard

Abstract: Despite increasing instances of machine translation (MT) systems including extrasentential context information, the evidence for translation quality improvement is sparse, especially for discourse phenomena. Popular metrics like BLEU are not expressive or sensitive enough to capture quality improvements or drops that are minor in size but significant in perception. We introduce the first of their kind MT benchmark testsets that aim to track and hail improvements across four main discourse phenomena: anaphora, lexical consistency, coherence and readability, and discourse connective translation. We also introduce evaluation methods for these tasks, and evaluate several competitive baseline MT systems on the curated datasets. Surprisingly, we find that the complex context-aware models that we test do not improve discourse-related translations consistently across languages and phenomena. Our evaluation benchmark is available as a leaderboard at <dipbenchmark1.github.io>. 

Under review as a conference paper at ICLR 2021
DIP BENCHMARK TESTs: EVALUATION BENCH- MARKS FOR DISCOURSE PHENOMENA IN MT
Anonymous authors Paper under double-blind review
ABSTRACT
Despite increasing instances of machine translation (MT) systems including ex- trasentential context information, the evidence for translation quality improve- ment is sparse, especially for discourse phenomena. Popular metrics like BLEU are not expressive or sensitive enough to capture quality improvements or drops that are minor in size but significant in perception. We introduce the first of their kind MT benchmark testsets that aim track and hail improvements across four main discourse phenomena: anaphora, lexical consistency, coherence and read- ability, and discourse connective translation. We also introduce evaluation meth- ods for these tasks, and evaluate several competitive baseline MT systems the curated datasets. Surprisingly, we find that the complex context-aware models that test do not improve riscaurse-related translations consistently across lan- guages and phenomena. Our evaluation benchmark is available as a leaderboard at pbinnhha 1 github.. iop.
INTRODUCTION AND RELATED WORK
The advances in neural machine translation (NMT) systems have led to great achievements in terms of state-of-the-art performance in automatic translation tasks. There have even been claims that their translations are no worse than what an average bilingual human may produce (Wu et al. (016)) or that the translations are on par with professional translators (Hassan et al.] 2018). However, extensive studies conducting evaluations with professional translators (Läubli et al. 2018, Popel et al. 2020) have shown that there a statistically strong preference for human translations in terms of fluency and overall quality when evaluations conducted monolingually or at the document level.
Document (or discourse) level phenomena (e.g., coreference, coherence) may not seem lexically significant, but contribute significantly readability and understandability of the translated texts (Guillou] 2012). Targeted datasets for evaluating phenomena like coreference (Guillou et al.) 2014] Guillou & Hardmeier, 2016 Baadhinova-Koltunnss
al. 2018 Bawden et 2018 Voita et al. 2018b), or ellipsis and lexical cohesion (Voita et al. 2019) have been proposed.
The NMT framework such as the Transformer (Vaswani et al., 2017 provides more flexibility to incorporate larger context. This has spurred a great deal of interest in developing context-aware NMT systems that take advantage of source or target contexts, e.g., e.g., Miculicich et (2018) Maruf
& Haffari 2018 8 Voita et al. (2018b 2019), Xiong et al. (2019), Wong et al. (2020). to name few.
Most studies only report performance on specific testsets, often limited to improvements in BLEU (Papineni et al. 2002). Despite being the standard MT evaluation metric, BLEU has been criticised its inadequacy; the scores are interpretable, and are not sensitive small improvements in lexical terms that may lead to big improvements in fluency or readability (Reiter) 2018). There is no framework for principled comparison of MT quality beyond mere lexical matching as done in BLEU: there are no standard corpora and no agreed-upon evaluation measures.
To address these shortcomings, we propose the DiP benchmark tests (for Discourse Phenomena), that will enable the comparison of machine translation models across discourse task strengths and source languages. We create diagnostic testsets for four diverse discourse phenomena, and also propose automatic evaluation methods for these tasks. However, discourse phenomena in transla- tions can be tricky to identify, let alone evaluate. A fair number of datasets proposed thus far have been manually curated, and automatic evaluation methods have often failed to agree with human

---

Under review as a conference paper at ICLR 2021
judgments (Guillou & Hardmeier 2018). To mitigate these issues, trained neural models for identifying and evaluating complex discourse phenomena and conduct extensive user studies to ensure agreements with human judgments. Our methods for automatically extracting testsets can be applied to multiple languages, and find cases that are difficult to translate without having to resort to synthetic data. Moreover, our testsets extracted in way that makes them representative of current challenges. They can be easily updated to reflect future challenges, preventing the pitfall of becoming outdated, which is a common failing of many benchmarking testsets.
We also benchmark established MT models on these testsets to convey the extent of the challenges they pose. Although discourse phenomena can and do occur the sentence-level (e.g. between clauses), we would expect MT systems that model extra-sentential context (Voita et al. 2018b] Zhang et al.] 2018; Miculicich et al.) 2018) to be more successful on these tasks. However, we observe significant differences in system behavior and quality across languages and phenomena, emphasizing the need for more extensive evaluation as a standard procedure. We propose to maintain leaderboard that tracks and highlights advances in MT quality that go beyond BLEU improvement.
Our main contributions this paper are as follows:
Benchmark testsets for four discourse phenomena: anaphora, coherence & readability, lexical consistency, and discourse connectives.
Automatic evaluation methods and agreements with human judgments.
Benchmark evaluation and analysis of four context-aware systems contrasted with baselines, for German/Rusian/Chinnee--- -hilsse---ennessh language pairs.
MACHINE TRANSLATION MODELS
Model Architectures. We first introduce the MT systems that we will be benchmarking on our testsets. We evaluate a selection of established models of various complexities (simple sentence- level to complex context-aware models), taking care to include both source- and target-side context- aware models. We briefly describe the model architectures here:
S2S: standard 6-layer base Transformer model Vaswann et al. 2017 which translates sentences independently.
CONCAT: A 6-layer base Transformer whose input is two sentences (previous and current tence) merged, with a special character as separator (Tiedemann & Scherrer 2017).
ANAPH: Voita al. (2018b) incorporate source context by encoding it with separate encoder, then fusing it in the last layer of a standard Transformer encoder using a gate. They claim that their model explicitly captures anaphora resolution.
TGTCON: To model target-context, we implement a version of ANAPH with an extra operation of multi-head attention in the decoder, computed between representations of the target sentence and target context. The architecture is described in detail in the Appendix (A.3).
SAN: Zhang et al. (2018)
source attention network: separate Transformer encoder encode source context, which is incorporated into the source encoder and target decoder using gates.
HAN: Miculicich et al. (2018) introduce hierarchical attention network (HAN) into the Trans- former framework to dynamically attend to the context at two levels: word and sentence. They achieve the highest BLEU when hierarchical attention is applied separately to both the encoder and decoder.
Datasets and Training. The statistics for the datasets used to train the models are shown in Ta- [I We tokenize the data using Jicba" Zh and Moses script for the other languages, lowercase the text, and apply BPE encoding from Sennrich et al. (0166) We learn the BPE encodings with the command 1 paarnjjoint www----e----oooeeeee and-v 8 40000. The scores reported are BLEU4, computed either through fairseq or NLTK (Wagner 2010). Further details about dataset composi- tion. training settings and hyperparameters can be found in the Appendix (A.7).
https///gtthbb..ommreee
2

---

Under review as a conference paper at ICLR 2021
BLEU scores. The BLEU scores the WM (De-En, Ru-En) and on the WMT-17 (Zh-En) testsets for each of the six trained models are shown in Table 2 We were unable to train HAN for Zh-En as the model was not optimized for training with large datasets. In contrast to increases in BLEU for selected language-pairs and datasets reported in published work, incorporating context within elaborate context-dependent models decreases BLEU scores for the Zh-En and De-En tasks. However, the simple concatenation-based model CONCAT performs better than S2S for De-En and Ru-En; this shows that context knowledge indeed helpful for improving BLEU.
Table 1: Statistics for different language pairs showing the number of (parallel) sentences in the train/dev/test datasets. The test data is from WMT- 14 for De-En and Ru-En, and WMT17 for Zh-En.
Table 2: BLEU scores achieved by bench- marked models on the WMT-14 (De-En, Ru-En) and the WMT-17 (Zh-En) testsets.
Model
De-Fn
Ru-Fn
Zh-Fn
Pair
Source
Train
Dev
Test
S2$
31.65
23.88
17.86
Zh-En IWSLT, Europarl, News, 17.195,748 UN, WikiTitles
2002
2001
CONCAT
31.96
24.56
17.17
ANAPH
29.94
27.66
16.31
ToTCON
29.94
26.06
15.67
De-En
IWSLT, Europarl, News
2,490,871
3,693
3,003
SAN
29.32
24.34
15.18
Ru-En
IWSLT, News
459,572
4,777
3,003
HAN
29.69
25.11
Table 3: MT output errors indicative of source texts with hard-to-translate phenomena (WMT-19).
Phenomena
Source
MT output
Reference
Anaphora
Der französische Rusfahrer war zur Krankenhaus gebliehen. Er wende im Laufe des Tages entlassen.
The French bus driver had remained the hospital.
The French bus driver had stayed in the hospital.
will be released over the day.
will released over the day.
Cohereuce
Pocrone виелороннии nepeneguyan-- -st norse сrолииооnmss arrronnfit.es nakcn.. Uponcmerrase eayrauzon, 29 cеиrιиōpu y.uue Hescrponaraa Auareetoo..
In Rostov, the SUV turned over taxi car. The procession uccurred on September 29 Nemirovic Street. Dunchenko.
SUV fliss after collision with
after collision
in Rastow. accident took
Reudability
place on September 29 on
Nemirovich- -ic---aec-DDeoooo street.
Lexical Consistency
in Landsham Grmddtticke Landsham-Sid Siid Nedbuigebiet Landshun--Sid
building fundshum land in landale south area of landsl-ssid
building in landsham properties south landshhmm south landshum
Discourse Connectives
3t #1 tee X.fi.g
share the future of China's is self- confidence China's commitment.
.shows the world China's 's confidence and responsibiiity open markets and share its future. It is fast to go alone but further to in crowds.
Walk Alone fasi, the crowd goes far.
3 BENCHMARK TESTSETS
We construct our benchmarking testsets based on four main principles:
Selectivity. The testsets need to provide hard to translate contexts for MT models. We ensure this by looking at translation errors made by system submissions to campaigns like WMT and IWSLT. Authenticity. The testsets cannot contain artificial or synthetic data but only natural text. Rather than generating testset samples using heuristics, we extract hard contexts from existing human- generated source text.
Multilinguality. The testset extraction method should be automatic and applicable to multiple lan- guages. Our framework can be used to extract testsets for all source languages that are part of the considered MT campaigns.
Adaptability. The testsets should be easy to update frequently, making them adaptable to improve- ments newer systems. Since we automatically extract hard contexts based on MT errors, our testsets are easy to update; they adapt to errors in newer (and possibly more accurate) systems, making the tasks harder over time.
We use the system outputs released by WMT and IWSLT for the most recent years (Nadejde et al.] 2016 Bojar et al. 2017 2018 2019 Cettolo et al. 2016 2017 to build our testsets. For De-En,
3

---

Under review as a conference paper at ICLR 2021
Ru-En and Zh-En, these consist of translation outputs from 68, 41 and 47 unique systems respec- tively. Since the data comes from a wide variety of systems, our testsets representatively aggregate different types of emrors from several (arguably SOTA) models. Also note that the MT models are benchmarking are not a part of these system submissions to WMT, so there is no potential bias in the testsets.
3.1 ANAPHORA
Anaphora are references entities that occur elsewhere in text; mishandling them result in ungrammatical sentences or the reader inferring the wrong antecedent, leading to misunderstanding of the text (Guillou 2012). We focus specifically on the aspect of incorrect pronoun translations.
Testset. To obtain hard contexts for pronoun translation, we look for source texts that lead to erro- neous pronoun translations in system outputs. We align the system translations with their references, and collect the cases in which the translated pronouns do not match the reference
Our anaphora testset is an updated version of the proposed by Jwalapuram et al. (2019). We filter the system translations based on their list of cases where the translations can be considered wrong, rather than acceptable variants. The corresponding source texts extracted test suite for pronoun translation. This gives us pronoun benchmark (estset of 2564 samples for De-En, 2368 for Ru-En and 1540 for Zh-En.
Evaluation. Targeted evaluation of pronouns in MT has been challenging as it is not fair to expect an exact match with the reference. Evaluation methods like APT (Miculieich Werlen & Popescu- Belis, 2017) or AutoPRF (Hardmeier & Federico 2010) are specific to language pairs or lists of pronouns, requiring extensive manual intervention. They have also been criticised for failing to produce evaluations that are consistent with human judgments (Guillou & Hardmeier 2018).
Jwalapuram et al.. (2019) propose a pairwise ranking model that scores "good" pronoun translations (like in the reference) higher than "poor" pronoun translations (like in the MT output) in context, and show that their model is good at making this distinction, along with having high agreements with human judgements. However, they do not rank multiple system translations against each other, which is our main goal; the absolute scores produced by their model are not useful since it is trained in pairwise fashion.
We devise a way to use their model to score and rank system translations in terms of pronouns. First, re-train their model with more up-to-date WMT data (more details in Appendix [.1).. We obtain a score for each benchmarked MT system (S2S, CONCAT, etc.) translation using the model, plus the corresponding reference sentence. We then normalize the score for each translated sentence by calculating the difference with the reference. To get an overall score for an MT system, the assigned scores are summed across all sentences in the testset.
Score Σ (r(ff)))) Pr(sys(8)
where pr(-10) denotes the score given to sentence by the pronoun model 6. The systems are ranked based this overall score, where a lower score indicates better performance. We conduct user study to confirm that the model rankings correspond with human judgments, obtaining an agreement of 0.91 between four participants who annotated 100 samples. Appendix A.I gives details (e.g.. interface, participants, agreement) about the study.
3.1.1 RESULTS AND ANALYSIS
The ranking results obtained from evaluating the MT systems on our pronoun benchmark testset using our evaluation measure are given in Table (first two columns). We also report common pronoun errors for each model based on our manual analysis (last three columns). Specifically, we observed the following types of errors in our analysis of a subset of the translation data:
(i) Gender copy. Translating from De/Ru to En often requires 'flattening' of gendered pronouns it, since De/Ru assign gender all nouns. In many cases, machine translated pronouns tend to (mistakenly) agree with the source language. For example, diese Wohnung in Earls Court..., und sie
*This process requires the pronouns in the target language he separate morphemes, in English.
4

---

Under review as a conference paper at ICLR 2021
Table 4: Pronoun evaluation: Rankings of the different models for each language pair, obtained from our evaluation procedure. Through manual analysis, % for the following types of errors are reported: Anaphora- instances Gender Copy, Named Entity and Language specific errors.
De-En
Ru-En
Za-Ea
Kank
Model
GC
Lang
Rank
Model
GC
NE
Rank
Model
GC
NE
CONCAT
55
35
11
HAN
31
48
21
1
ANAPH
0
80
20
12
SAN
27
46
3
CONCAT
20
25
528
HAN
44
22
33
SAN
32
44
24
4
ToTCrNN
10
90
ANAPII
42
17
41
5
$25
37
26
SAN
33
67
ToTCON
hatte... is translated : apartment Earls Court, and she had..., which keeps the female gender expressed in sie, instead of translating it to it.
(ii) Named entity. A particularly hard problem to infer gender from a named entity, egg. Lady Liberty....he is meant to...- she is wrongly translated to it. Such examples demand higher inference abilities (e.g. distinguish male/female names).
(iii) Language specific phenomena. In Russian and Chinese, pronouns are often dropped sen- tences become ungrammatical English without them. Pronouns can also be ambiguous in the source language; e.g., in German, the pronoun sie can mean both she and you, depending capi- talization, sentence structure, and context.
Overall, we observe that the advantages of contextual models are not consistent across languages. They seem to use context well Ru-En, but fail to outperform S2S or CONCAT in De-En, while Zh-En is inconclusive. The TGTCON model is consistently poor in this task. The partial success of the S2S model can explained by its tendency to use it as the default pronoun, which statistically appears most often due to the lack of grammatical gender in English. More variability in pronouns occurs in the outputs of the context-aware models, but this does not contribute to a greater success.
3.2 COHERENCE AND READABILITY
Pitler Nenkova 2008) define coherence as the ease with which a text can be understood, and view readability as an equivalent property that indicates whether it is well-written.
Testset. To test for coherence and readability, we try to find documents that can considered hard to translate. We use the coherence model proposed by Moon et al. (2019), which is trained in pairwise ranking fashion on WSJ articles, where a negative document is formed by shuffling the sentences of an original (positive) document. It models syntax, inter-sentence coherence relations and global topic structures. It has been shown in some studies that MT outputs are incoherent Läubli al. 2018). We thus re-train the coherence model with reference translations as positive and MT outputs as negative documents to better capture the coherence issues that present in MT outputs (more details in Appendix A.2).. We use older WMT submissions from 2011-2015 to ensure that the training data does not overlap with the benchmark testset data.
(mmith al. 2015 2016;
The coherence model takes system translation (multi-sentential) and its reference as input and produces score for each. Similar to Eq. D we consider the difference between the scores produced by the model for the reference and the translated text as the coherence score for the translated text.
For given source text (document) in the WMT testsets, we obtain the coherence scores for each of the translations (i.e.. WMT//WSLT submissions) and average them. The source texts are sorted based on the average coherence scores of their translations. The texts that have lower average co- herence scores can considered to have been hard to translate coherently. We extract the source texts with scores below the median. These source texts form our benchmark testset for coherence and readability. This yields 272 documents (5,611 sentences) for De-En, 330 documents (4,427 sentences) for Ru-En and 210 documents (3,050 sentences) for Zh-En.
Evaluation. Coherence and readability is also a hard task to evaluate, as it can be quite subjective. We resort to model-based evaluation here as well, to capture the different aspects of coherence in translations. We use our re-trained coherence model to score the benchmarked MT system transla- tions and modify the scores for use in the same way the anaphora evaluation (Eq. T to obtain
5

---

Under review as a conference paper at ICLR 2021
relative ranking. As mentioned before (Sec. B). the benchmarked MT systems do not overlap with the WMT system submissions, so there is no potential bias in evaluation since the testset extrac- tion and the evaluation processes independent. To confirm that the model does in fact produce rankings that humans would agree with, and to validate our model re-training, we conduct a user study, and obtain agreement of 0.82 between three participants who annotated 100 samples. More details about the study can be found in the Appendix (A.2).
3.2.1 RESULTS AND ANALYSIS
Table 5: Coherence and Read- ability evaluation: Rankings of the different models for each language pair, obtained from our evaluation procedure.
We identified some frequent coherence and readability errors (more examples in Appendix (A8):
(i) Inconsistency. As in (Somasundaran et al. 2014), we observe that inconsistent translation of words across sentences (in partic- ular named entities) breaks the continuity of meaning.
De-En
Ru-En
Zh-En
(i)) Translation error. Errors various levels spanning from un- grammatical fragments to model hallucinations introduce phrases which bear little relation to the whole text (Smith et al. (016):
CONCAT S25 ANAPH
ANAPH ToYCON
ANAPH
S25
SAN
ToTCoN
ToTCON
S2S
SAN
Reference: There is huge applause for the Festival Orchestra, who appear on stage for the first time - in casual leisurewear in view of the high heat.
Translation: There is great applause for the solicitude orchestra, which is on the stage for the first time, with the heat once again in the wake of an empty leisure clothing.
From the rankings in Table [] we can see that ANAPH is the most coherent model for Zh-En and Ru-En but performs poorly in De-En, similar to the pronoun benchmark. Generally CONCAT is better than complex contextual models in this task.
3.3 LEXICAL CONSISTENCY
Lexical consistency in translation was first defined as 'one translation per discourse' by Carpuat 22009), Le., the translation of partieular source word consistently the same target word in that context. [uullou (2013) analyze different human-generated texts and conclude that human transla- tors tend to maintain lexical consistency support the important elements text. The consistent usage of lexical items in a discourse can be formalized by computing the lexical chains (Morris & Hirst
1991 Lotfipour-San Saedi 1997).
Testset. To extract a testset for lexical consistency evaluation, we first align the translations from the system submissions with their references. In order to get reasonable lexical chain formed by consistent translation, we consider translations of blocks of 3-5 sentences in which the (lemmatized) word we are considering occurs least twice in the reference. For each such word, we check the corresponding system translation produces the same (lemmatized) word at least once, but fewer such cases, the system translation
than the number times the word occurs in the reference. has failed to be lexically consistent in translation (see Table 3 for example). We limit the errors considered to nouns and adjectives. The source texts these cases form the benchmark testset. This gives testset with 618 sets (i.e., text blocks) for De-En (3058 sentences), 732 sets for Ru-En (3592 sentences) and 961 sets for Zh-En (4683 sentences).
Evaluation. For lexical consistency, we adopt simple evaluation method. For each block of 3-5 sentences, we either have consistent translation of the word in focus, or the translation is inconsis- tent. We simply count the instances of consistency and rank the systems based on the percentage. Model translations are considered lexically inconsistent if at least one translation of particular word matches the reference translation, but this translated word occurs fewer times than in the reference. For samples where no translations match the reference, we cannot be sure about inconsistency, since a synonym of the reference translation could have been used consistently. Therefore, we not con- sider them for calculating the percentage used for the main ranking, but we report the consistency percentage as fraction of the full testset for comparison (further discussion Appendix (A.3)

---

Under review as a conference paper at ICLR 2021
Table 6: Lexical consistency evaluation: Rankings of the different models for each language pair, ranked by the % of samples that are Consistent compared to inconsistent samples, then by % of consistent samples in the Full testset. Also shown are the results of manual error analysis on a subset of the translations for Synonyms, Related words, Omissions, Named Entity, Random translation.
Do-Ea
Ru-Bi
Rk
Model
Con
%Tull
Syn
Rei
Omn
NE
Rd
Rk
Model
Con
%FFul
Syn
Rei
Om
NE
Rd
CONDAT
42.83
34.30
15
23
10
TemCCN
26
11.88
20
8
28
28
16
TaTCON
42.57
32.36
13
27
7
27
27
2
ANAPH
23,99
12.16
15
0
26
15
54
SAN
41.52
30.90
38
19
24
14
3
CONCAT
23.81
12.29
15
8
15
18
44
ANAPH
40.72
30.90
46
21
SAN
19.22
9.42
9
18
42
$25
40.00
525
18.99
9.29
21
21
Zii-En
Rk
Model
Con
%uFull
Syn
Rel
Om
NE
Rd
ANAPH
30.94
17.48
43
14
2
TeTCoN
29.84
16,02
14
29
43
14
CONCAT
28.17
16.65
66
33
SAN
26.33
13.42
38
25
25
3.3.1 RESULTS AND ANALYSIS
The rankings of the MT systems based the percentage of samples with consistent translations on the lexical consistency benchmark testsets are given Table (first four columns), along with findings from manual analysis subset of the translations (last five columns). Our manual inspection of the lexical chains shows the following tendencies:
(i) Synonyms & related words. Words are exchanged for their synonyms (poll survey), hyper- nyms/hyponyms (ambulance car) or related concepts (wine vineyard).
(ii) Named entities. Models tend to distort proper names and translate them inconsistently. For example, the original name Fücchtorf (name of town) gets translated to feeding-communiyy
(iil) Omissions. Occurs when words are omitted altogether from the lexical chain.
The overall low quality of Russian translations contributes the prevalence of Random translations, and the necessity to transliterate named entities increases NE errors for both Ru-En and Zh-En. Here we see some complex contextual models performing well; TGTCON leads the board across De-En, Ru-En and Zh-En En, with ANAPH performing similarly well for Ru-En and Zh-En. Generally, we should be seeing a consistent advantage for target-side context models, which should be able to "remember" their own translation of word from previous sentences; however this only materializes for TGTCON and not for HAN.
3.4 DISCOURSE CONNECTIVES
Discourse connectives are used to link the contents of texts together by signaling coherence rela- tions that essential to the understanding of the texts (Prasad al.) (2014). Failing to translate a discourse connective correctly can result in texts that are hard to understand or ungrammatical. Finding errors discourse connective translations can be quite tricky, since there are often many acceptable variants. To mitigate confusion, we limit the errors we consider in discourse connectives the setting where the reference contains connective but the translations fail produce any (see Table for an example).
User Study. To confirm that missing connectives are problematic, we conduct user study. Partici- pants are shown two previous sentences from the reference for context, and asked to choose between two candidate options for the sentence that may follow. These options consist of the reference trans- lation which includes connective, and MT output that is missing the connective translation. Participants are asked to choose the sentence which more accurately conveys the intended meaning. See Figure 4b in Appendix A.4 for example interface.
We obtaín an agreement of 0.82 between two participants who annotated 200 samples, that trans- lations with connectives preferred. If the MT outputs with missing connectives structured in such a way as to have implicit discourse relations, the agreements that favoured the references should be significantly lower. However, the strong agreements favouring the reference with the

---

Under review as a conference paper at ICLR 2021
Table Discourse connective evaluation: Rankings of the different models for each language pair, ranked first by their Accuracy and then by the percentage where ANY connective is produced. Each set of rankings followed by the results of the manual analysis on subset of the translation data for Omissions, Synonyms, Mistranslations.
Di
Ri-lin
Zh: Ei
Rank
Medel
ANY
Mis
Rank
Model
Ace
ANY
Ous
Stn
Mis
Rank
Madell
ANY
Om
Syn
Mis
Asscni
49.42
75.55
15
AWARK
40.81
65.30
535
55.24
80.66
43
42
14
SAN
72.67
$25
37:41
73.47
TOTC
52.84
83.06
44
26
30
22
$25
73.54
60.54
CONDAT
46.77
9693
CONCENT
12.66
70.06
Raa
68.18
69.76
72
76
0
RAN
31.39
55.10
76
31
nective indicate that the missing connectives in MT outputs are indeed an issue. More details about the study can be found in the Appendix (A.4).
Testset. It would not be appropriate to simply extract connectives using list of candidates, since those words may not always act in the capacity of a discourse connective. In order to identify build simple explicit connective classifier (a neural model) using
the discourse connectives, annotated data from the Penn Discourse Treebank or PDTB (Prasad et al. (018)) (details in Appendix A.4).. The classifier achieves an average cross-validation Fl score of 93.92 across the 25 sections of PDTBv3, proving that it generalizes well.
After identifying the explicit connectives in the reference translations, align them with the cor- responding system translations and extract the source texts of cases with missing connective trans- lations. We only use the classifier on the reference text, but consider all possible candidates in the system translations to give them the benefit of the doubt. This gives us discourse connective benchmark testset with 172 samples for De-En, 147 for Ru-En and 362 for Zh-En.
Evaluation. There has been some work on semi automaaitoiiccc evaluation of translated discourse con- nectives (Meyer et al. 2012 Hajlaoui & Popescu-Belis 2013): however, limited to only En-Fr, based on dictionary list of equivalent connectives, and requires using potentially noisy alignments and other heuristics. the interest of evaluation simplicity, we expect the model to produce the same connective the reference. Since the nature of the challenge is that connectives tend to be omitted altogether, we report both the accuracy of connective translations with respect to the reference, and the percentage of cases where any candidate connective is produced.
3.4.1 RESULTS AND ANALYSIS
The rankings of MT systems based on their accuracy of connective translations are given in Ta- ble 7, along with our findings from a manual analysis on a subset of the translations. In benchmark outputs, we observed mostly omissions of connectives (disappears in the translation). synonymous translations (e.g., Naldo is also a great athlete on the bench Naldo's 's "great sport" on the bank, too.), and mistranslations. More examples can be found in the Appendix (A.8).
The ranking shows that the S2S model performs well for Ru-En and Zh-En but not for De-En. ANAPH continues its high performance in Ru-En and this time also De-En, doing poorly for Zh-En, while HAN is consistently poor with a lot of omissions.
DISCUSSION
Our benchmarking re-emphasizes the gap between BLEU scores and translation quality the dis- course level. The overall BLEU scores for De-En and Ru-En are higher than the BLEU scores for Zh-En; however, we see that Zh-En models have higher accuracies in the discourse connective task, and also outperform Ru-En in lexical consistency. Similarly, for Ru-En, both SAN and HAN have higher BLEU scores than the S2S and CONCAT models, but are unable to outperform these simpler models consistently in the discourse tasks, often ranking last.
We also reveal a gap in performance consistency across language pairs. Models may be tuned for particular language pair, such as ANAPH trained for En-Ru (Voita al.) (018a). For the same language pair (Ru-En), we show results consistent with what is reported; the model ranks first or second for all phenomena. However, it is not consistently successful in other languages, e.g., ranking
8

---

Under review as a conference paper at ICLR 2021
close to bottom for almost all cases in De-En. In general, findings match the conclusions from Kim et al. (2019) regarding the lack of satisfactory performance gains in context-aware models.
Although testsets and evaluation procedures have their limitations, like only checking for miss- ing connectives or being unable to detect consistently translated synonyms of reference translations, they are first step toward a standardized, comprehensive evaluation framework for MT models that spans multiple languages. They are useful for measuring basic model proficiency, performance consistency and for discovering MT deficiencies. Discourse-aware models have been advocated for improving MT (Sennrich 2018); as more models are proposed, our framework will be a valuable resource that provides a better picture of model capabilities. With advances in NMT models and also in evaluation models for complex phenomena, harder challenges can be added and evaluated.
5 GENERALIZABILITY TO OTHER LANGUAGES
Procedures used to create our testsets can be generalized to create testsets in other languages. We briefly describe the possibilities here:
The pronouns need to be separate morphemes (and not attached to verbs etc.). If there are several equivalent pronoun translations, list may be needed they be excluded from being considered translation errors; e.g., Miculicich Werlen & Popescu-Belit (2017) has such list for French, list can also be collected through user studies as in Jwalapuram et al. (2019).
Anaphora:
Coherence & Readability: The coherence model (Moon et al. 2019) used to find poorly trans- lated texts was re-trained on reference vs. MT outputs. It is also possible to this for other languages for which WMT (or IWSLT) system outputs are available. The coherence model from Moon et al. (2019) is an end-to-end neural model that does not rely on any language-specifie fea- tures, and thus can be trained any target language. However, language-specific multilingual coherence models could also used since Moon et al. (2019) primarily train and test their model on English (WSJ) data.
Lexical Consistency: lemmatizer was used to reduce common suffixes for detecting lexical consistency (e.g., "box" and "boxes" should not be detected as inconsistent words), similar tool will be needed for any other target language; e.g., CLTK (Johnson et al. 2014-2020) provides lemmatizer for several languages.
Discourse Connectives: Discourse connectives also need be separate morphemes. We built a classifier trained on PDTB data to identify connectives since they are ambiguous in English. Datasets analogous to PDTB in other languages e.g., PCC (German) (Bourgonje & Stede 2020)
and CDTB (Chinese) (Zhou & Xue 2015), etc. are available.
6 CONCLUSIONS
We presented the first of their kind discourse phenomena based benchmarking testsets called the DiP tests, designed to be challenging for NMT systems. Our main goal emphasize the need for comprehensive MT evaluations across phenomena and language pairs, which we do by highlighting the performance inconsistencies of complex context-an aware models. We will release the discourse benchmark testsets and evaluation frameworks for public use, and also propose to accept translations from MT systems maintain leaderboard for the described phenomena.
REFERENCES
Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. Evaluating discourse phe- nomena in neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technolo- gies, Volume 1 (Long Papers), pp. 1304-1313, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/II1 org/ htts//athloooyKKK--111
URL https: www aclweb.
Ondrej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Had- dow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, and Julia Kreutzer (eds.). Pro-

---

Under review as a conference paper at ICLR 2021
ceedings of the Second Conference Machine Translation, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/ URL https:
/www www.b.llwebcoo org/ www.aa..wbb.orry..aaaooooo/o/./.
Ondfej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor (eds.). Pro- ceedings of the Third Conference on Machine Translation: Research Papers, Brussels, Belgium, October 2018. Association for Computational Linguistics. h www aclweb. org/ hogg/anhoolggy//i/ 6300
Ondiej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor (eds.). Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), Florence, Italy, August 2019. Association for Computational Linguistics. URLhttps://wwr
aclweb. wwtp://///n.nch.///aae..... 5200
Peter Bourgonje and Manfred Stede. The potsdam commentary corpus 2.2: Extending annotations for shallow discourse parsing. In LRECC, 2020.
Marine Carpuat. One translation per discourse. In SEW@NAACL-HLT, 2009.
Mauro Cettolo, Christian Girardi, and Marcello Federico. Wit Web inventory of transcribed and translated talks. In Proceedings of the 16th Conference of the European Association for Machine Translation (EAMT), pp. 261-268, Trento, Italy, May 2012.
Mauro Cettolo, Niehues Jan, Stüker Sebastian, Luisa Bentivogli, R. Cattoni, and Marcello Federico. The iwslt 2016 evaluation campaign. 2016.
Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Niehues Jan, Stüker Sebastian, Sudoh Kat- suitho, Yoshino Koichiro, and Federmann Christian. Overview of the iwslt 2017 evaluation cam- paign. 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 11Xiv:1810.04805, 2018.
Liane Guillou. Improving pronoun translation for statistical machine translation. In EACL, 2012.
Liane Guillou. Analysing lexical consistency in translation. In Proceedings of the Workshop on Dis- course in Machine Translation, pp. 10-18, Sofia, Bulgaria, August 2013. Association for Com- putational Linguistics. URL https: /www. ac web.
Liane Guillou and Christian Hardmeier. PROTEST: A test suite for evaluating pronouns in ma- chine translation. In Proceedings of the Tenth International Conference on Language Re- sources and Evaluation (LREC' 16), pp. 636-643, Portorož, Slovenia, May 2016. European Lan- guage Resources Association (ELRA). URL htt.https:: wwwp///wwww/////////////////ww ac web. www.we/w./w...oooooogg/goooo ogy/ L16 -6100
Liane Guillou and Christian Hardmeier. Automatic reference-based evaluation of pronoun transla- tion misses the point. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP '18. pp. 4797-4802, Brussels, Belgium, 2018
Liane Guillou, Christian Hardmeier, Aaron Smith, Jörg Tiedemann, and Bonnie Webber. ParCor 1.0: A parallel pronoun- corefernne corpus to support statistical MT. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC" 14), pp. 3191- 3198, Reykjavik, Iceland, May 2014. European Language Resources Association (ELRA). URL http:// www. rec- conf htt.a//..........cooooo/e///e//a Paper. pdff
Najch Hajlaoui and Andrei Popescu- Belis. Assessing the accuracy of discourse connective transla- tions: Validation of an automatic metric. In CICLing, 2013.
10

---

Under review as a conference paper at ICLR 2021
Christian Hardmeier and Marcello Federico. Modelling pronominal anaphora in statistical machine translation. In Proceedings of the 2010 International Workshop on Spoken Language Translation, IWSLT 10, pp. 283-289, Paris, France, 2010.
Hany Hassan, Anthony Aue. Chang Chen, Vishal Chowdhary, Jonathan R. Clark, Christian Fed- ermann, Xuedong Huang. Marcin Junczys-Dowmun William Lewis, Mu Li, Shujie Liu, T. M. Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. Achieving human parity on automatic chinese to english news translation. ArXiv, abs/ 80388003.5555 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):
1735-1780, November 1997. ISSN 0899-7667.
Kyle Johnson, Patrick Burns, John Stewart, and Todd Cook. Cltk: The classical language toolkit, 2014-2020.
hwt.tttww...ooocccoooommmmceeete cltk/cltkt
Prathyusha Jwalapuram, Shafiq Joty, Irina Temnikova, and Preslav Nakov. Evaluating pronominal anaphora in machine translation: An evaluation measure and test suite. In EMNLP-IJNNLP, 2019.
Yunsu Kim, Thanh Tran, and Hermann Ney. When and why is document-level context useful in neural machine translation? ArXiv, abs/1910.00094, 2019.
Ekaterina Sopshinova-Koltunkki Christian Hardmeier, and Pauline Krielke. ParCorFull: paral- lel corpus annotated with full coreference. In Proceedings of the Eleventh International Con-
ference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018.
European Language Resources Association (ELRA). URL https:/ /twp///wwww.////
org/
anthology/L18 1065]
Samuel Läubli, Rico Sennrich, and Martin Volk. Has machine translation achieved human parity? a case for document-level evaluation. In EMNLP, 2018.
Kazem Lotfipou-Saedd Lexical cohesion and translation equivalence. 1997.
Sameen Maruf and Gholamreza Haffari. Document context neural machine translation with mem- ory networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1275-1284, Melbourne, Australia, July 2018. Asso- ciation for Computational Linguisties. URL https: http::///www....lebbbb.
aclweb. org/anthology/
P18-1118
Thomas Meyer, Andrei Popescu-Belis N. Hajlaoui, and Andrea Gesmundo. Machine translation of labeled discourse connectives. In AMTA 2012, 2012.
Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. Document-level neural machine translation with hierarchical attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2947-2954, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. URL https: w/www.aclwwt org/ http://nnhhhoooooooeeeoeeee 13355
Lesly Miculicich Werlen and Andrei Popescu-Belis. Validation of an automatic metric for the ac- curacy of pronoun translation (APT). In Proceedings of the Third Workshop on Discourse in Machine Translation, WMT 17, pp. 17-25, Copenhagen, Denmark, 2017.
Han Cheol Moon, Tasnim Mohiuddin, Shafiq R. Joty, and Xiaofci Chi. A unified neural coherence model. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Process- ing and the 9th International Joint Conference on Natural Language Processing, pp. 2262-2272, 2019, URL https: www. aclweb. org/
1231. .pf
Jane Morris and Graeme Hirst. Lexical cohesion computed by thesaural relations an indicator of the structure of text. Computational Linguistics, 17:21 144, 1991.
Maria Nadejde, Alexandra Birch, and Philipp Koehn. Proceedings of the first conference on machine translation, volume 1: Research papers. The Association for Computational Linguistics, 2016.
11

---

Under review as a conference paper at ICLR 2021
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Anmual Meeting of the Association for Computational Linguistics, ACL '12, pp. 311-318, Philadelphia, PA, USA, 2002.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, NAACL-HLT" 18, pp. 2227-22222222 New Orleans, LA, USA, June 2018.
Emily Pitler and Ani Nenkova. Revisiting readability: unified framework for predicting text quality. In EMNLP, 2008.
M. Popel, M. Tomková, Tomek, Lukasz Kaiser, Jakob Uszkoreit, Ondrej Bojar, and Z. Zabokrt- sky. Transforming machine translation: a deep learning system reaches news translation quality comparable human professionals. Nature Communications, 11, 2020.
Rashmi Prasad, Bonnie L. Webber, and Aravind K. Joshi. Reflections the penn discourse tree- bank, comparable corpora, and complementary annotation. Computational Linguistics, 40:921- 950, 2014.
Rashmi Prasad, Bonnie L. Webber, and Alan Lee. Annotation in the pdtb The next generation. 2018.
Ehud Reiter. A structured review of the validity of BLEU, Computational Linguistics, 44(3):393- 401, 2018.
Rico Sennrich. Why the time is ripe for discourse in machine translation. http://homepages. inf. ed. ac wwt.kk//eeennni/////mm20000
paf 2018
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016)., Berlin, Germany., 2016.
Karin Sim Smith, Wilker Aziz, and Lucia Specia. A proposal for coherence corpus in machine translation. In DiscoMT EMNLP, 2015.
Karin Sim Smith, Wilker Aziz, and Lucia Specia. The trouble with machine translation coherence. In Proceedings of the 19th Annual Conference of the European Association for Machine Translation, pp. 178-189, 2016. URL httllttps//// /www ac www.ww.wbbb..................... hwtp:/o.rggaaaathhoaaoooooo antho woatholooy//ii 6 6407
Swapna Somasundaran, Jill Burstein, and Martin Chodorow. Lexical chaining for measuring dis- course coherence quality in test-taker essays. In Proceedings of COLING 2014, the 25th In- ternational Conference on Computational Linguistics: Technical Papers, pp. 950-961, Dublin, Ireland, August 2014. Dublin City University and Association for Computational Linguisties.
URL https::////........ //t:://wwwww wwwww/ww.ommmmmom..wwol//. acl web. wwebborgg/arttooloy//C4441100000
Jörg Tiedemann and Yves Scherrer. Neural machine translation with extended context. In Proceed- ings of the Third Workshop Discourse in Machine Translation, pp. 82-92. Copenhagen, Den- mark, September 2017. Association for Computational Linguistics. doi: 10. 11.188//1/W11--8811
URL httpssts::/ /wwww a web. org/ htw.w/www.....tttoooooo//...//
Jörg Tiedemann. Parallel data, tools and interfaces in opus. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Mehmet Ugur Dogan, Bente Maegaard, Joseph Mar- iani, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12) Istanbul, Turkey, may 2012. European Lan- guage Resources Association (ELRA). ISBN 978-2-9517408-777,
12

---

Under review as a conference paper at ICLR 2021
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser. and Illia Polosukhin. Attention is all you need. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp- 5998-6008. Curran Associates, Inc., 2017. URI. p
Elena Voita, Pavel Serdyukov. Rico Sennrich, and Ivan Titov, Context-aware neural machine trans- lation learns anaphora resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACI. 11, pp. 1264-1274, Melbourne, Australia, 201 Sa.
Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. Context-aware neural machine trans- lation learns anaphora resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1264-1274 1274, Melbourne, Australia, July 2018b. Association for Computational Linguistics. URL httpp: / www. ac web org/ anthology/PIB-1111
Elena Voita, Rico Sennrich, and Ivan Titov. When a Good Translation is Wrong in Context: Context- Aware Machine Translation Improves on Deixis, Ellipsis. and Lexical Cohesion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 2019. Association for Computational Linguistics. 05979. pd
W. Wagner. Steven bird, ewan klein and edward loper: Natural language processing with python, analyzing text with the natural language toolkit. Language Resources and Evaluation, 44:421- 424, 2010.
KeyYen Yen Wong, Sameen Maruf, and Gholamreza Haffari. Contextual neural machine translation improves translation of cataphoric pronouns. In ACL, 2020.
Yonghui Wa, Mike Schuster, Zhifeng Chen, Quoe V. Le. Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin John- son, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang. Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals. Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. ArXiv, 000166699.11444... 2016.
H. Xiong. Zhongjun He, Hua Wu, and H. Wang. Modeling coherence for discourse neural machine translation. In AAAI, 2019.
Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. Improving the transformer translation model with document-level context. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 533-542, Brussels, Belgium, October-Novembot 2018. Association for Computational Linguistics. doi: 10.18653/v/DD18-1044
Y. Zhou and N. Xue. The chinese discourse treebank: a chinese corpus annotated with discourse relations. Language Resources and Evaluation, 49:397-431, 2015.
Michat Ziemski, Marcin Dowryss,D and Bruno Pouliquen. The united nations parallel cor- pus v1.0. Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France, may 2016. European Language Resources Association (ELRA). ISBN 978-2-9517408-9-1L
A APPENDIX
13

---

Under review as a conference paper at ICLR 2021
A.1
ANAPHORA
Source: Germen
Content:
Car Sageing on Laurs Segemund endete in Ulenetreee von tae dee Bittraglleee-- and Nitt gns vepasce duch en 1.6, grgin Horic rug yo Row le cas
The not dny, they set oectinn tox the 2e Fates the moet demanding wetton secing them dimbt arourd st0 not abttude seettor mn heat, and then destmn again. They curnol ruvard beroousing topether swirmaay pord.
hetrinls dymppscheon fenralamars Exstichs Sragemund, Richenprattemee
Vergitich
Rundan kraftioe.
Candidate
monestery church St. went the finat stage Preiburg.
Slegemund, who had been plagued back prodiemms was abla to get sircherge
only ance and was powertioos comparions the presious rounds.
Condidate
After the morring impulse on the fourth pligrimage day in the impresshre monzstery church St. Pater, we went stage Fraiburg.
Candidats
plagued
effect
playrimage Proibarg.
Condidate
Sigemund, who has been struggiing with baak protlems,, was only alde hold her serve once, and acomed powerfess in compartson earlier rounds.
Figure I: User study interface (bilingual setup) for pronoun translation ranking. Pronouns the sen- tences are highlighted in red.
User study interface (monolingual setup) for pronoun translation ranking. Pro- nouns in the sentences are highlighted in red.
Figure 2:
Detailed Steps for Testset Generation. Steps that were used to generate the anaphora test set:
We aligned the references with various MT system outputs from WMT & IWSLT cam- paigns.
2. Given list of pronouns (a closed set), we find the cases where the pronoun translations in the references do not match the aligned, corresponding pronoun translations the MT system outputs.
3. Jwalapuram et al. (2019) provide a list of pronoun pairs that are not equivalent (non- interchangeable) translations of each other. We filter the cases where the mismatched pronoun translations could be equivalent translations.
4. The corresponding source texts of the remaining wrongly translated cases are added to our test set.
Evaluation Model Description. Jwalapuram al. (2019) build pairwise ranking model that trained to distinguish "good" pronoun translations (like the reference) from "had" pronoun translations (like in erroneous MT outputs) in context. We briefly describe the model architecture:
1. The model first obtains contextual representations of sentence (reference MT output) by attending over a common context of two previous sentences from the reference.
2. The pronoun representations from this sentence are extracted, and these in turn attend over the contextual sentence representations. This results in contextual representations of the pronouns in the sentence.
3. These contextual pronoun representations are passed on to the final layer which generates the scores.
4. The model is trained with a pairwise ranking objective representations higher than the erroneous MT pronoun representations.
the reference pronoun
We believe that the model scores are both sensitive and specific to pronouns, for several reasons:
Sensitivity, Results from the original paper are reported for:
Reference MT output, where the model is 90.69% accurate.
Reference vs. noisy reference, where everything in the noisy version is identical ex- cept for the pronoun; here the accuracy is 89. %..
High agreements (>=0.8) with human judgments on similar noisy data.
Specificity. The model scores only originate from pronouns because:
The final layer of the model only pronoun representations to generate the scores.
14

---

Under review as a conference paper at ICLR 2021
Table Results of the re-trained pronoun scoring model.
Training data
Test data
Accuracy
WMT13-18
WMT-19
86.76
Attention heat maps from/Jwaaapp Walapuram et al. (2019) show that the scores are influenced only by the pronouns in the sentence, both for the reference vs. noisy and the reference vs. system translation cases.
We also conduct user study to confirm agreement between the model scores and human judgments, which is described later in this section.
Re-trained model. The pronoun evaluation model results reported in Jwalapuram al. (2019) based on a model that is trained on WMT1 1-111-15 data and tested on WMT--001 data. We re-train the model with more up-to-date data from WMT13-18, and test the model WMT-11 data. Note that this training data is taken from WMT submissions, which do not overlap with the benchmarked MT models; there is therefore no conflict using this trained model evaluate the benchmarked model translations. Results are shown in Table g Their model scores the translations in context; we provide the previous two sentences from the reference translation as context according to their settings.
User study. To confirm that Normalization-basee ranking of systems agrees with human judg- ments, we conducted a user study. Participants are asked to rank given translation candidates in terms of their pronoun usage. We include the reference in the candidates,
control.
We ask participants to rank system translations directly rather than a synthetically constructed con- trastive pair (as was done by Jwalapuram al. (2019)) to ensure that our evaluations, which will be conducted on actual translated texts, are reliable.
We first conducted the study in bilingual setup, in the presence of the source for German-English Participants were shown a source context of two sentences and the source sentence in bold, followed by three candidate translations of the source sentence, one of which the reference. The other two were translations with different pronoun errors produced by MT systems. 3 participants annotated 100 such samples. The bilingual (German- (herman-Fnglish) user study interface for pronoun translation ranking is shown in Figure
We then conducted the study monolingual setup, i.e., native speakers are shown the reference context in English, and the two candidate English translations and the reference translation pos- sible options for the sentence that follows (Figure 2 participants annotated 100 such samples.
The results are analysed to check how often (i) the reference is preferred over the system translations (our control), and (ii) the users agree in preference over the system translations (i.e., human judgment for translation quality).
Due to the nature of the dataset, annotators are more likely to choose the reference as the better candidate, which yields skewed distribution of the annotations; traditional correlation measures such as Cohen's 's kappa are not robust to this, and thus for this and all subsequent studies, we report the more appropriate Gwet's ACI/gamma coefficient. It is also the agreement reported by Dwalapu- ram et al. (2019). The control yielded a Gwet's AC1 agreement of 0.72 for the bilingual study and 0.82 for the monolingual study. The agreements higher for the monolingual study: Läubli et al. (2018) also find that human judgements differ in monolingual setups.
We evaluate the rankings from our modified evaluation method with the monolingual study data and obtain an agreement of 0.91, justifying the use of our modified pronoun model for evaluation.
The total assigned scores (difference between the reference and translation scores) ob- tained for each system after summing the over the samples in the respective testsets are given in Table 9 The models ranked based these scores from lowest score (least difference compared to reference; best performing) to highest score (highest difference compared to reference; worst performing).
Results.
15

---

Under review as a conference paper at ICLR 2021
Table Models ranked according to their performance (best to worst) in anaphora according the evaluation model, with BLEU score for comparison. Model scores given here are obtained by sub- tracting the score for the model translation from the score for the reference translation, and summing the score differences across the dataset. Hence, smaller model scores indicate better performance (closer to reference scores).
De-En
Ru-En
Zh-En
Rk
Model
BLEU
Score
Rk
Model
BLEU
Score
Rk
Model
BLEU
Score
CONCAT
31.96
112.583
HAN
25.11
160.411
ANAPH
16.31
43.930
2
S2S
31.65
113.783
ANAPH
27.66
164.603
2
CONCAT
17.17
49.092
SAN
29.32
117.838
CONCAT
24.56
168.092
3
S2S
17,86
64.176
HAN
29.69
118.067
SAN
24.34
176.143
4
TGTCON
15.76
111,683
ANAPH
29.94
129.662
S2S
23.88
181.887
5
SAN
15.18
123,908
IGTCON
29.94
131.699
TGTCON
26.06
183.969
Candidatt Ac Britinn
Thursday masch apritt for mas The Briscs sye brows the rirst the recorta in qualficstion and than wes Zasland Great
Olymmicc
brokin again the round
recard holder from Niw Zeslandary, leaving
medale.
Candidate Br
tack cycling team tnioe Oltmaae recard and veine sprint.
first of emplc corracted British, via recormecidd fiom Zealanders. The Britin tram with New Zaala met in sprine Chympic recard ownere the young. lnavingg them
first round computition
Candidatt
The Seitun vack cydiig toam motic reaard tyice right und wio thennass The bettan truck upnnt rose Zouland, Settan wold flve
Thersday Leboe the Ohrmals and
Boolendera Symiet maa dritan Zeolanders the Ohympic nourd hokoer of the Wit. kocking with sihrr readeb.
Figure 3: User interface for coherence study. The participants are shown 4- 4-sentence texts and asked to rank them in terms of coherence and readability.
A.2 COHERENCE
Re-trained model. We re-train the pairwise coherence model in Moon et al. (2019) to suit the MT setting, with reference translations as the positive documents and the MT outputs the negative documents. The results are shown in Table
Table 10: Results of the re-trained coherence model.
Training data
Test data
Accuracy
WMT11-15
|WM17-188
77.35
User study. Figure 3 shows our user study interface. The participants are shown three candidate same source text, and asked to rank the texts on how coherent and readable they are. To optimize annotation time, participants are only shown the first four sentences of the document; they annotate 100 such samples. We also include the reference as one of the candidates for control, and to confirm that are justified in re-training the evaluation model to assign higher score to the reference. Three participants took part in the study. Our control experiment results in an AC1 agreement of 0.84.
English translations
The agreement between the human judgements and the rankings obtained by using the original coherence model trained permuted WSJ articles (also news domain, like the WMT data), is 0.784. The fact that the original model performs no worse than 0.784 shows that there are definitely coherence issues in such (MT output vs reference) data that are being picked up.
The agreement between the human judgements and the retrained coherence evaluation model's 's rank- ings is 0.82. Our re-treined model is therefore also learning useful task-specific features in addition
16

---

Under review as a conference paper at ICLR 2021
Table 11: Models ranked according their performance (best worst) in coherence according our evaluation, with BLEU for comparison. Coherence scores given here are obtained by subtracting the score for the model translation from the score for the reference translation, and summing the score differences across the dataset. Hence, smaller model scores indicate better performance (closer to reference scores).
De-En
Ru-En
Zh-En
Rk
Model
BLEU
Score
Rk
Model
BLEU
Score
Rk
Model
BLEU
Score
CONCAT
31.96
5038.057
ANAPH
27.66
7412.280
1
ANAPH
16.31
2599.465
29.32
CONCAT
24.56
7641.254
2
CONCAT
17.17
2709 448
3
S2S
31.65
5120.633
3
TGTCON
26.06
7742.092
3
S28
17.86
3545.707
4
ANAPII
29.94
5166.320
SAN
24.34
7875.095
4
TGTCON
15.76
4440 890
5475.636
23.88
8133.746
5
SAN
15.18
4538.013
HAN
29.69
5516.808
HAN
25.11
8719.370
general coherence features. The high agreement validates our proposal the modified coher- ence model to evaluate the benchmarked MT systems.
The total assigned scores (difference between reference and translation scores) obtained for each system after summing the over the samples in the respective testsets given in Table The models are ranked based on these scores from lowest score (best performing) to highest score (worst performing).
Results.
A.3 LEXICAL CONSISTENCY
Dataset extraction. One possible issue with our method could be that reference translations may contain forced consistency, i.e,, human translators introduce consistency to make the text more read- able, despite inconsistent word usage in the source. It may not be reasonable to expect consistency in a system translation if there is none the source. To confirm, we conducted a manual analysis where compared the lexical chains of nouns and adjectives in Russian source texts against the lexical chains the English reference. We find that a majority (77%) of the cases, the lexical chains in the source are reflected accurately in the reference, and there are relatively few cases where humans force consistency.
Evaluation. is possible that the word used in the system translation not the same as the word in the reference, but the MT output is still consistent (e.g., synonym used consistently). We tried to use alignments coupled with similarity obtained from ELMo (Peters et al. 2018) and BERT (Devlin ett al. 2018) embeddings evaluate such cases to avoid unfairly penalizing the system translations, but we found this to be noisy and unreliable. Thus, we check consistency against the reference; at least one translation matches the reference but the translated word occurs fewer times than it does in the reference, the translation is considered inconsistent. For samples where there is no common translation between the system output and the reference, we cannot be sure if it consistent or not, so we exclude it for calculating the primary ranking percentage. We therefore report both the percentage of consistent samples as a fraction of consistent + inconsistent samples, and percentage of consistent samples as a fraction of the full dataset for comparison purposes.
A.4 DISCOURSE CONNECTIVES
Table 12: Results of the connective classification model.
Training data
Test data
Precision Recall F1 score
PDTBv3 Sections 0-24 PDTBv3 Sections 0-24 (avg CV)
95.58
92.35
93.92
Connective Classification model. We build an explicit connective classifier to identify candidates that are acting in the capacity of discourse connective. The model consists of LSTM layer (Hochreiter & Schmidhuber (997) followed by a linear layer for binary classification, initialized by
EL embeddings (Peters al. 2018). We annotated data from the Penn Discourse Treebank
17

---

Under review as a conference paper at ICLR 2021
Context:
He longs to keep singing, He's hightor.
Candidate A:
keep repeating song swnnt forget them," Karel admitted.
Candidate
"I keep repeating song lyrics so won't forge: them," Karel adm ttod.
(a) Reference vs. reference with deleted connective
Context:
Elke Twesten must immediately give up her seat". said nationel heed and cempaign leader the Greens Ricdaktionskezwwert Deutschland (RND). The the CDU represents clear distortion of the voters' will.
Candidate A:
"This all sighs 5PD-rampaigner, Friday morning, had news reached the party headquarters.
Candidate B:
"It just can't true, sighed campaigner, when bad news party headgu on Friday afternon..
(b) Reference vs. system translation with missing connective
Figure 4: User study interfaces for discourse connectives.
(PDTBv3) PPasad et al. 2018) and conduct cross-validation experiments across all 25 sections. The average Precision, Recall and F1 scores from the cross-validation experiments
reported in Table 12 Our classifier achieves an average cross-validation FI of 93.92, which shows that generalizes very well. The high precision also provides certainty that the model is classifying discourse connectives reliably.
User Study. To confirm that the presence of the connective conveys information and contributes to the readability and understanding of the text, we conducted two user studies. For the first study, as presented in Figure 4a participants are shown two previous sentences from the reference for context, and asked to choose between two candidate options for the sentence that may follow. These options consist of the reference translation with the connective highlighted, and the same text with the connective deleted.
Participants are asked to choose the sentence which more accurately conveys the intended meaning. There were participants who annotated 200 such samples. The reference with the connective was chosen over the version without the connective with an ACI agreement of 0.98. Table [3 shows the connective-wise breakdown.
In the second study, the participants were shown the reference along with the system translation that was missing the connective (Figure 4b). this study, the setup has no artificially constructed data; the idea is to check if there is possibility that the system translation is structured in such a way as to require no connective. However, the ACI agreement for preferring the reference was 0.82 (2 annotators for 200 samples; different annotators from the first study) for this study as well, which still quite high. Table [3 has the connective-wise breakdown; here we see that the results are slightly different for certain connectives, but overall the strong preference for the reference with the connective is retained. Our assumption that connectives must be translated validated through both studies.
Note that participants may not prefer the version without the connective due to loss of grammaticality or loss of sense information. Although indistinguishable in this setting, argue that since both affect translation quality, it is reasonable to expect a translation for the connectives.
Note that for both studies, participants were also given options to choose 'Neither' in case they didn't 't prefer either choice, or 'Invalid' in case there was an issue with the data itself (e.g., transliteration issues, etc.); data that marked such was excluded from further consideration.
18

---

Under review as a conference paper at ICLR 2021
Study I: Reference vx. Connective Deleted Reference
Study 2: Reference vs. Missing Connective Translation
Connective
ACI Age.
Ref
# Nuiisy
Tie
Connective
ACI Agr.
Ref
#
TTe
und
0.96
136
11
und
0.84
127
20
26
also
1.0
35
3
18
also
0.82
36
5
by
1.0
D
1.
0
40.38
0.79
1.0
sO
1.0
because
1.0
10
0
because
1.0
0
then
dien
0.57
with
if
if
1.0
0
thes
1.0
0
tlus
1.0
D
indeed
indeed
without
1.0
0
without
1.0
U
unless
1.0
0
unless
1.0
0
subsequenlly
subsoquently
1.0
0
ultimately
0
ultimatelyy
1.01
0
before
before
proviously
previously
once
once
bowever
1.0
0 0
however
Table 13: Connective-wise results for the connective user studies. The table also shows the number of times the Reference System translation was chosen (summed for both annotators). The Tie column shows the number of times the users showed preference. Note that ties not included in the agreement. Other samples not included were the ones marked as invalid by the annotators due misalignment errors, severe grammatical issues, etc.
Table 14:
Discourse phenomena: Anaphora (restricted to anaphoric pronouns), Lexical Consistency, and Discourse Connectives in popular NMT datasets (for English). The column ANY shows the proportion of sentences which contain any of the listed phenomena.
Dataset
Anaph.
Lex. Con.
Conn.
ANY
UN
0%
31%
0%
31%
Europar
17%
24%
12%
49%
News Commentary
5%
18%
18%
37%
IWSLT
11%
19%
32%
42%
A.5 TGTCON MODEL ARCHITECTURE
Here we describe the model architecture for TGTCON. The decoder introduced Vaswani et al. (2017) is used for encoding the target sentence and the encoder adopted from the original encoder of the Transformer architecture is used to encode context of the target side. Each part, target decoder and context encoder, repeated 6 times (N=6). In the last layer, two multi-head attention operations are performed, followed by layer normalization (similar to Vaswani et al. (2017)). The first multi- head attention the self-attention on target sentence, whereas the second multi-head attention is cross attention between representation of target and target context. These two representations are fused by gated linear sum which decides the amount of information from each representation that to be passed on. showw the model architecture in detail.
A.6 MODEL. TRAINING
Training Data It is essential to provide the models with training data that contains adequate amounts of discourse phenomena, if we expect them to learn such phenomena. To construct such datasets, we first manually investigated the standard WMT corpora consisting of UN (Ziemski al.) 2016), Europarl (Tiedemann) 2012) and News Commentary, as well as the standard IWSLT dataset (Cettolo et al. 2012).. We analyzed 100 randomly selected pairs of consecutive English sentences from each dataset, where the first sentence was treated as the context. 14shhos the percentage of cases containing the respective discourse phenomena.
19

---

Under review as a conference paper at ICLR 2021
Context enniched enooder (tom Vetaa of
Crscee contect entiched decoder
Feed Farner
Mac HUNDO
Atstto
ME-HbE ABr
Adciioms
((N-1)
Mantac Hutione
lagel
Target
Figure 5: TGTCON model architecture.
In accordance with intuition, data sources based on narrative texts such as IWSLT exhibit increased amounts of discourse phenomena compared strictly formal texts such the UN corpus. On the other hand, the UN corpus consists of largely unrelated sentences, where only lexical consistency is well-represented due to the usage of very specific and striet naming of political concepts. We decided to exclude the UN corpus and combine the other datasets that have more discourse phenomena for De-En and Ru-En; for Zh-En, we keep UN and add WikiTitles to bolster the BLEU scores. Our training dataset is therefore combination of Europarl, IWSLT and News Commentary datasets, plus UN and WikiTitles for Zh-En. The development set combination of WMT-2016 and older WMT data (excluding 2014). Note that our validation set does not have any data in common with the benchmark testsets. We test on WMT-2014 (De/Ru-En//WMT-2) ( data. We tokenize the data using Jieba for Zh and the Moses softward for the other languages, lowercase the text, and apply BPE encodings from Sennrich et al. (2016). We learn the BPE encodings with the command 40000.
learn- jointt bpe-and-vocab
Training For the context-aware models, we use the implementations from official author reposi- tories. As the official code for ANAPII (Voita et al. 2018b) has not been released, implement the model in the Fairseq framework (Ott et al. 2019 For training the S2S and CONCAT models, we used the Transformer implementation from fairseq. We confirmed with the authors of HAN and SAN that our configurations were correct, and we took the best configuration directly from the ANAPH paper.
A.7 MODEL PARAMETERS
Parameters used train HAN displayed in Table 15 and parameters for the S2S, CONCAT, ANAPH, and SAN models are displayed Table 16
A.8 ERROR EXAMPLES
Examples for the different types of errors encountered across the tasks are given in Table I7
20

---

Under review as a conference paper at ICLR 2021
Table 15: Configuration parameters for training HAN model, taken from the authors' repository
ht attpss:/ /ttt::://ituu..... com/ idiap/HAN NMTH
Parameters
Values
Step 1: sentence-level NMT
-encoder_type
transformer transformer
-decoder_type
-enc_layers
6
-dec_ dlye_laeers
6
-label_ Sobbll_soothing
0.1
-rn size
512
-position_encodiiieg
-rrooout
0.1
-batch_size
4096
start_decay__g
20
-epochs
20
-max geaeoxggreettrr__ batches
16
-batch_ type
tokens
-noraalization
tokens
-accum_count
4
-optim
adam
adam_beta)
0.998
-decay_method
noam
-warmup_stepa
8000
-learning rate
2
-max_grad _ord__ormm
-param_init
0
-param init iniot_gllrotttttttt__ggo_oo_o_tt
-trii__paat sentences
Step 2: HAN encoder
others see Step I
others
see Step
-batch_sizz
1024
-start_ caart_decay_at
2
-epochs
10
-max generator batches
32
-train_ part
all
-context_type
HAN HAN_enc _nc
-context_size
3
Step 3: HAN joint
others see Step 1
others see Step I
-batch_size
1024
start decay_
2
-epochs
-max_ Laaa__generttr__tg geatches
32
-train_part
all
context__yppe
HAN join 3
-context_sis
-train from
[HAN enc model]
21

---

Under review as a conference paper at ICLR 2021
Table 16: Configuration parameters for training SAN, ANAPH, CONCAT, S2S models. Parameters of ANAPH are taken from the original paper Voita et al. 2018h) and parameters of SAN are taken from the authors' repository: https: : /ttgggtttb//.//// com THUNLP-MYT -wt..//t////./uuunn--- Document-Trans and user manual for the THUMT library which provides the basic Transtormer model: https: github. com/ THUNLP MT TW/OTTHOTT/TT
Maseaamattteeeeeeaaauaaaaaa pdf Parameters which are not listed were left default. Note that the max update for Zh-En was set to 400,000 due to the larger dataset size.
Model
Parameters
Values
SAN
Stepl: sentence-level
batch_sir
6250
update_cycle
4
train_stepp
200000
Step 2: context-aware Transformer
num _oin_coneett_cc layers
ANAPH
-optimizer
adam
adam-betas
'(0.9, 0.98)
clip-norm
0.0
Ir-scheduler
inverse_sqn
warmup-init-Ir
le-07
warmup-updates
4000
-Ir
0.0007
-min-Ir
le-09
-criterion
smoothed cross _aossseetroop
label-smoothing
0.1
weight-decay
0.0
max-tokens
1024
update-freq
32
share-all-embeddings
max-update
100000
CONCAT
-optimizer
adam
adam-betas
'(0.9, 0.98)
clip-norm
0.0
Ir-scheduler
inverse sqrtt
warmup-init-Ir
1e-07
warmup-updater
4000
-Ir
0.0007
-min-Ir
lc-09
criterion
aabelsmmotheel cross entropy _rosssentrrpy
label-smoothing
0.1
weight-decay
0.0
max-tokens
4096
update-freq
8
share-all-embeddings
-mx-update
100000
S2S
as in CONCAT
as in CONCAT
22

---

Under review as a conference paper at ICLR 2021
Table 17: Examples for the types of errors found in the translations. S: denotes source, T: denotes model translations while R: denotes reference translations.
Phenomenon
Example
Anaphora
Gender Copy
S: Mir wurde diese Wohnung Earls Court gezeigt, und sie hatte ...
T: shown this apartment in Earls Court, and she had
Named Entity
T: Lady Liberty stepping forward. It meant
carrying torch of liberty R: She mcant to be carrying the torch of Liberty.
steppinngforward
Language Specific
S: Auftraggeber: Napoleon.. the pronoun ihr refers to the noun Karten (English: maps).
The German pronoun ihr can mean her, their, or your.
T: (..) detailed maps for towns and municipalities (...). Your contructor Napoleon.
R: (.) detailed maps for towns and municipalities (...). Their commissioner. Napoleon.
Lexical Consistency
Synonym
T: Watch the Tory party conference. The convention is supposed to be about foreign policy, (...).
R: Under tight security the Tory party conference. The party conference was to address foreign policy (...).
Related Word
T: In the collision of the car with taxi, 27-year-old passer was fatally injured.
R: A 27-year old passenger was fatally injured when the ambulance collided with a taxi.
Named Entity
T: The Preding-Communice farmer, however, also had the ready-filled specialities.
The demand for the good "made in Feed orf" was corre correspondingly high. R: But the Füchtorf farmer also had bottled specialtics with him.
There was a
demannd the good "made Füchtorf beverage.
Omission
T: (...) during the single-family home attempt, stayed by the royal highlands thanks to the burglar alarm.
They got off when culprits turned hand Friday just before 20 a.m.
R: It is thanks to the alarm system that the attempt in the Königswieser Straße the single family home (...).
On Friday just before 20,00 the alarm when the offenders took action.
Coherence
Ungrammatical
T: "They didn't 't play badly for long periods like Stone Hages. , like Hip Horst Senser.
Only the initial phase, we ve been totally wasted", annoyed the ASV coach.
R: "Over long periods, they had as in Steinhagen, as ugainst Hüllhorst not played badly. We only overslept the initial phase". said the ASV coach annoyed.
Hallucination
T: Before appointing Greece Jeffrey Pyett was the US ambassador to Kiew.
When it came to the Maidan and the coup in 2014, it was a newspaper. R: Before his appointment, Geoffrey Ross Pyatt was an ambassador Kiyv. During mission, the Maydan events and state coup happened, reminds Gazeta.Ru
Inconsistency
T: The me-in-house airline crashed on Sunday afternoon at a parking near Essen-Mosquitos. Essen Mill is small airport that's used by airline pilots.
R: On Sunday afternoon, the single-seated aireraft crashed parking lot the airport Essen-Mülheem Essen-Müüheins
small airport. which is frequently used by pilots with light private planes.
Discourse Connectives
Omission
T: Two people died driving their against
R: Two people died after driving their car into tree.
Synonym
T: Naldo's "great sport" on the bank,
R: Naldo is also great athlete on the bench
Mistranslation
T: Gfk's leadership departs from disappointing business figures
R: GfK managing director steps down after disappointing figures
23


Reviews:
[{"review_id": "dVzaLfSHXFU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose specific test sets for document-level NMT. They target anaphora, coherence/readability, lexical consistency and discourse connectives, and are available for multiple language pairs. The first two challenge sets rely on model-based evaluation.\n\nStrengths:\n\nThe test sets, which target various discourse phenomena, directly evaluate the output of the models (contrarily to some existing multiple-choice challenge sets).\n\nThe types of mistakes made by NMT models are manually examined.\n\nThe authors validate the quality of their metrics by comparing against human judgements (although the number of samples is arguably small).\n\nWeaknesses:\n\nThe evaluated NMT models all date from 2018 or earlier.\n\nThe anaphora challenge sets are only a minor update over previous work.\n\nAll language pairs use English as the target language.\n\nOther remarks and questions:\n\nFor the anaphora and coherence/readability test sets, future work may \"cheat\" by using the evaluation models as part of the NMT systems.\n\nIs normalizing the scores actually useful? The reference scores should be the same across all systems, so it only shifts all results by a constant and doesn't affect relative performance.\n\nWhats steps would be needed to construct similar test sets for En->X language pairs?", "extra_scores": {}, "extra_text": {}}, {"review_id": "VNZPADm6OYl", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a dataset, a trained evaluation metric and a leaderboard for evaluating discourse phenomena for machine translation. They test this on a range of discourse level translation models and develop metrics which evaluate the models according to their performance on four discourse phenomena: anaphora, lexical consistency, coherence and readability, and discourse connective translation. \n\nStrengths:\nThis paper delivers multiple contributions which could have significant impact on the field of discourse level machine translation. They release data for three language pairs and using their method one could extend it relatively easily into others. I like the thoughtful way that the authors find examples of hard discourse phenomena and each phenomena requires distinct handling. \n\nWeaknesses:\nThey rely on previous work to create the Anaphora test set and evaluation model (Jwalapuram et al. (2019)). They should have explained at a high level how the Jwalapuram evaluation model works and they should have given a general idea of the rules used to filter the anaphora test set: how many rules, an example, would this be possible to do for other languages or does it only work well for English? \nI would have liked more discussion about the extensibility of their approach into languages other than English. \nThey should have used sacrebleu or at least specified if the BLEU scores were tokenised and true cased. \n\nI think this paper should be accepted because of combined strength of the dataset/metric/leaderboard. I think fine grained evaluation of hard phenomenon is the way forward for improving already very good MT models. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7x_I6x6OCqX", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents four methods for creating benchmarking datasets, each focusing on a particular discourse phenomenon which is difficult/hard to solve with existing context-aware neural machine translation (NMT) models.  It also evaluates several existing NMT models using the created data and makes a remark about their current status, such as the superiority of one method in one task.  This work has the originality and I am happy to know that some researchers devote their efforts to address these issues.  Moreover, I am sure that the field of NMT definitely benefits from this work, once the created data are made publicly available.\n\nHowever, I am very skeptical that the contribution of this paper fits for ICLR, since the methodology presented in this paper itself mainly consists of a pile of human efforts.  In other words, if I understand the contents properly, only the technical advancement presented in this paper is TgtCon, a variation of the anaphora-centric method proposed by Voita et al. (2018b).  However, it cannot be a substantial merit to the community, since it does not necessarily perform better than existing methods and the reason of its deterioration is not analyzed.\n\nConsidering the main focus of the paper, i.e., creating benchmarking datasets, the paper should rather fit for a journal article in the field of natural language processing, where the authors can give more details in creating the datasets, such as procedure, tool, and attributes of annotators, not in appendices, and more careful analyses of the results, including WHY each method does (not) perform well on a particular discourse phenomenon.\n\nIndeed, when I was reading this paper, I suffered from the fact that the main part of the paper is not self-contained.  For instance, the proposed TgtCon is not explained in Section 2 but the readers are advised to see an appendix.  Other information that is indispensable in data creation but missing in this paper is the detail of human judgment, such as the proficiency of the evaluators, protocol, and judgment criteria.  Each section reports on agreement ratio, but not Cohen's kappa.  With these reasons, I am not completely convinced of the quality of the resulted datasets and the portability of the proposed methods.\n\nBelow shows some questions.\n\nQ1. Is there a reason to choose these four particular discourse phenomena?  Are they exclusive to each other?\n\nQ2. Test sets for coherence/readability and discourse connectives are significantly smaller than those for the other two phenomena.  I am not sure that a test set with 200 or less examples is enough.\n\nQ3. In Section 3.2.1, the authors regard the inconsistency of named entities.  However, in many error classification schemes, terminology errors are, irrespective of incorrect translations and inconsistency, distinguished from coherence errors.  For instance, the Multidimensional Quality Metrics (*1) locate them in different branches.\n\n(*1) http://www.qt21.eu/mqm-definition/\n\nQ4. What does \"Random translations\" in Section 3.3.1 mean?", "extra_scores": {}, "extra_text": {}}, {"review_id": "2Nu_Yw-0EK4", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a benchmark for discourse phenomena in machine translation. Its main novelty lies in the relatively large scale, spanning three translation directions, four discourse phenomena, and 150-5000 data points per language and phenomenon. A relatively large number of systems from previous work is benchmarked on each test set, and agreement with human judgments is measured.\n\npositives:\n\n- clearly, a lot of thought and effort has gone into the creation of the benchmarks, and this is the most diverse set of benchmarks for discourse phenomena in MT so far. \n\n- extensive experiments with models from previous work, along with human analysis.\n\nnegatives:\n\n- two of the four benchmarks, anaphora and coherence, are evaluated by neural models trained on WMT outputs, so the interpretation of scores is opaque, and their validity is unclear. Specifically, Jwalapuram et al. (2019) train a neural network to distinguish references from MT output based on the ELMo representations of pronouns, but in principle, this model can use signals other than the correctness of pronouns translation to make this distinction. Similarly, the model by Moon et al. (2019) was originally trained to distinguish real documents from randomly shuffled ones, and I can see how their complex neural network would then learn to rely on coherence features. However, this submission uses reference translations as positive, MT output as negative examples, so it again may learn to use features other than coherence for its decisions.\n\n- also, I'm not fully convinced about the validity of the automatic discourse connective evaluation. According to the manual analysis, there is a large proportion of false negatives (synonymous translations flagged as errors), and rankings would change if synonyms were counted as correct. I was also not satisfied with the evidence that the omission of connectives is generally an error. The human study was a bit simplistic in that it just deleted connectives (although more changes might be needed to ensure grammaticality) or used noisy MT output rather than alternative human references. It also seems to have been monolingual. If the test set consists of examples with ambiguous or implicit discourse relations in the source, then it may actually be the right translation strategy to omit them. I'm worried that a benchmark that rewards explicitation and punishes leaving discourse relations implicit may set the wrong incentives.\n\n- I was surprised by the low results (already in terms of BLEU) of some of the tested variants. Authors describe in great care their efforts to fairly reproduce previous work, including the use of original code and hyperparameters where possible, but I can't help but think that the models are suboptimally trained, and that statements about whether context-aware models consistently improve discourse phenomena are tainted by this. \n\nrecommendation:\n\nI'm leaning negative on the current version of the paper and benchmark. I think the test sets have been carefully assembled, and along with the various types of models evaluated on them, this work has value. But before I'd recommend that the benchmarks actually be used in the field, authors would need to improve upon the evaluation scores used for anaphora, coherence, and discourse connectives and make sure they really are targeted towards the phenomena they claim to measure, and do not have large blind spots.\n\nfurther questions and minor problems:\n\n- did you perform early stopping? What was your stopping criterion?\n\n- do you have an explanation why anaphora scores differ wildly between systems for ZH-EN (table 9)? \n\n- the discourse connectives test sets are based on examples where the reference contains a connective, but MT output does not. What MT system was used? Do the respective source segments generally contain explicit discourse connectives, or are the respective discourse relations generally implicit in the source?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a dataset and a trained evaluation metric for evaluating discourse phenomena for MT. Several context-aware MT models are compared against a sentence level baseline. The paper develops metrics which evaluate the models according to their performance on four discourse phenomena: anaphora, lexical consistency, coherence and readability, and discourse connective translation. Data is released for three language pairs (all using English as the target language). 

First, I’d like to point out that creating datasets and benchmarks for analyzing/evaluating discourse-level errors in machine translation is an extremely valuable contribution. This paper is addressing a very relevant problem and even though there is no new model/method/algorithm being proposed, this work *fits* this conference - it is my opinion that the community should welcome and value more than it currently does the efforts spent in creating high quality datasets that can help make progress in the field.

There was substantial discussion among reviewers about this paper. 

The main weaknesses raised by the reviewers were:
- Limited information about the process to create the anaphora test, which was a contribution of prior work (Jwalapuram et al. (2019) - this was addressed in the updated version; but the anaphora challenge sets seem to be only a minor update over previous work.
- All language pairs use English as the target language, and it is not simple to extend this approach beyond English target languages.
- Lack of detail on how BLEU scores were computed (tokenised? true cased? My recommendation is to use sacrebleu) - this was clarified in the rebuttal.
- The evaluated NMT models all date from 2018 or earlier.
- Two of the 4 benchmarks (anaphora and coherence) are evaluated by neural models trained on WMT outputs, which makes the interpretation of scores is opaque, and their validity is unclear.

While the creation of a benchmark for discourse evaluation of MT is a laudable effort as mentioned above, it is my opinion that due to some of the weaknesses above the current version of this work is not yet ready for publication. However, I strongly encourage the authors to improve upon these points and resubmit their work to another venue. I list some suggestions below to improve this paper.

My biggest concern with the current version is the last weakness above. As pointed out by a reviewer, the framework of Jwalapuram et al. (2019) provides empirical support for the model's sensitivity (if there is a pronoun error, does the metric pick it up?). But they don’t necessarily capture model *specificity* (if the metric ranks one output higher, can we be confident that this is because of a pronoun translation error?). For the coherence metric, authors make an argument that their metric is sensitive to coherence issue, but concerns remain about whether it is sufficiently specific to these issues. In the rebuttal, authors argue that BLEURT is sentence-level, but they could easily aggregate sentence-level judgments and report correlation between BLEURT and human coherence  judgments to show that their metric correlates better with human coherence judgments than BLEURT or even just BLEU. Besides BLEURT, I would add there are other recently proposed metrics that may capture discourse phenomena (neural metrics trained against MQM annotations or sentence-level human assessments with document context) and should be compared against: check COMET [1] or PRISM [2] (the latter is sentence-based but could be adapted for paragraphs or documents).

There is also prior work comparing various context-aware machine translation approaches against a sentence-level baseline, some with negative findings [3,4,5]. I suggest the authors look at this related work in future iterations of their paper.

[1] https://arxiv.org/pdf/2009.09025.pdf

[2] https://arxiv.org/pdf/2004.14564.pdf 

[3] https://www.aclweb.org/anthology/2020.eamt-1.24.pdf

[4] https://arxiv.org/pdf/1910.00294.pdf 

[5] https://www.aclweb.org/anthology/2020.emnlp-main.81.pdf 

Decision: Reject