DiP Benchmark Tests: Evaluation Benchmarks for Discourse Phenomena in MT

1 INTRODUCTION AND RELATED WORK . The advances in neural machine translation ( NMT ) systems have led to great achievements in terms of state-of-the-art performance in automatic translation tasks . There have even been claims that their translations are no worse than what an average bilingual human may produce ( Wu et al. , 2016 ) or that the translations are on par with professional translators ( Hassan et al. , 2018 ) . However , extensive studies conducting evaluations with professional translators ( Läubli et al. , 2018 ; Popel et al. , 2020 ) have shown that there is a statistically strong preference for human translations in terms of fluency and overall quality when evaluations are conducted monolingually or at the document level . Document ( or discourse ) level phenomena ( e.g. , coreference , coherence ) may not seem lexically significant , but contribute significantly to readability and understandability of the translated texts ( Guillou , 2012 ) . Targeted datasets for evaluating phenomena like coreference ( Guillou et al. , 2014 ; Guillou & Hardmeier , 2016 ; Lapshinova-Koltunski et al. , 2018 ; Bawden et al. , 2018 ; Voita et al. , 2018b ) , or ellipsis and lexical cohesion ( Voita et al. , 2019 ) , have been proposed . The NMT framework such as the Transformer ( Vaswani et al. , 2017 ) provides more flexibility to incorporate larger context . This has spurred a great deal of interest in developing context-aware NMT systems that take advantage of source or target contexts , e.g. , Miculicich et al . ( 2018 ) , Maruf & Haffari ( 2018 ) , Voita et al . ( 2018b ; 2019 ) , Xiong et al . ( 2019 ) , Wong et al . ( 2020 ) , to name a few . Most studies only report performance on specific testsets , often limited to improvements in BLEU ( Papineni et al. , 2002 ) . Despite being the standard MT evaluation metric , BLEU has been criticised for its inadequacy ; the scores are not interpretable , and are not sensitive to small improvements in lexical terms that may lead to big improvements in fluency or readability ( Reiter , 2018 ) . There is no framework for a principled comparison of MT quality beyond mere lexical matching as done in BLEU : there are no standard corpora and no agreed-upon evaluation measures . To address these shortcomings , we propose the DiP benchmark tests ( for Discourse Phenomena ) , that will enable the comparison of machine translation models across discourse task strengths and source languages . We create diagnostic testsets for four diverse discourse phenomena , and also propose automatic evaluation methods for these tasks . However , discourse phenomena in translations can be tricky to identify , let alone evaluate . A fair number of datasets proposed thus far have been manually curated , and automatic evaluation methods have often failed to agree with human judgments ( Guillou & Hardmeier , 2018 ) . To mitigate these issues , we use trained neural models for identifying and evaluating complex discourse phenomena and conduct extensive user studies to ensure agreements with human judgments . Our methods for automatically extracting testsets can be applied to multiple languages , and find cases that are difficult to translate without having to resort to synthetic data . Moreover , our testsets are extracted in a way that makes them representative of current challenges . They can be easily updated to reflect future challenges , preventing the pitfall of becoming outdated , which is a common failing of many benchmarking testsets . We also benchmark established MT models on these testsets to convey the extent of the challenges they pose . Although discourse phenomena can and do occur at the sentence-level ( e.g. , between clauses ) , we would expect MT systems that model extra-sentential context ( Voita et al. , 2018b ; Zhang et al. , 2018 ; Miculicich et al. , 2018 ) to be more successful on these tasks . However , we observe significant differences in system behavior and quality across languages and phenomena , emphasizing the need for more extensive evaluation as a standard procedure . We propose to maintain a leaderboard that tracks and highlights advances in MT quality that go beyond BLEU improvement . Our main contributions in this paper are as follows : • Benchmark testsets for four discourse phenomena : anaphora , coherence & readability , lexical consistency , and discourse connectives . • Automatic evaluation methods and agreements with human judgments . • Benchmark evaluation and analysis of four context-aware systems contrasted with baselines , for German/Russian/Chinese-English language pairs . 2 MACHINE TRANSLATION MODELS . Model Architectures . We first introduce the MT systems that we will be benchmarking on our testsets . We evaluate a selection of established models of various complexities ( simple sentencelevel to complex context-aware models ) , taking care to include both source- and target-side contextaware models . We briefly describe the model architectures here : • S2S : A standard 6-layer base Transformer model ( Vaswani et al. , 2017 ) which translates sentences independently . • CONCAT : A 6-layer base Transformer whose input is two sentences ( previous and current sentence ) merged , with a special character as a separator ( Tiedemann & Scherrer , 2017 ) . • ANAPH : Voita et al . ( 2018b ) incorporate source context by encoding it with a separate encoder , then fusing it in the last layer of a standard Transformer encoder using a gate . They claim that their model explicitly captures anaphora resolution . • TGTCON : To model target-context , we implement a version of ANAPH with an extra operation of multi-head attention in the decoder , computed between representations of the target sentence and target context . The architecture is described in detail in the Appendix ( A.5 ) . • SAN : Zhang et al . ( 2018 ) use source attention network : a separate Transformer encoder to encode source context , which is incorporated into the source encoder and target decoder using gates . • HAN : Miculicich et al . ( 2018 ) introduce a hierarchical attention network ( HAN ) into the Transformer framework to dynamically attend to the context at two levels : word and sentence . They achieve the highest BLEU when hierarchical attention is applied separately to both the encoder and decoder . Datasets and Training . The statistics for the datasets used to train the models are shown in Table 1 . We tokenize the data using Jieba1 for Zh and Moses scripts2 for the other languages , lowercase the text , and apply BPE encodings3 from Sennrich et al . ( 2016 ) . We learn the BPE encodings with the command learn-joint-bpe-and-vocab -s 40000 . The scores reported are BLEU4 , computed either through fairseq or NLTK ( Wagner , 2010 ) . Further details about dataset composition , training settings and hyperparameters can be found in the Appendix ( A.7 ) . 1https : //github.com/fxsjy/jieba 2https : //www.statmt.org/moses/ 3https : //github.com/rsennrich/subword-nmt/ BLEU scores . The BLEU scores on the WMT-14 ( De-En , Ru-En ) and on the WMT-17 ( Zh-En ) testsets for each of the six trained models are shown in Table 2 . We were unable to train HAN for Zh-En as the model was not optimized for training with large datasets . In contrast to increases in BLEU for selected language-pairs and datasets reported in published work , incorporating context within elaborate context-dependent models decreases BLEU scores for the Zh-En and De-En tasks . However , the simple concatenation-based model CONCAT performs better than S2S for De-En and Ru-En ; this shows that context knowledge is indeed helpful for improving BLEU . 3 BENCHMARK TESTSETS . We construct our benchmarking testsets based on four main principles : Selectivity . The testsets need to provide hard to translate contexts for MT models . We ensure this by looking at translation errors made by system submissions to campaigns like WMT and IWSLT . Authenticity . The testsets can not contain artificial or synthetic data but only natural text . Rather than generating testset samples using heuristics , we extract hard contexts from existing humangenerated source text . Multilinguality . The testset extraction method should be automatic and applicable to multiple languages . Our framework can be used to extract testsets for all source languages that are part of the considered MT campaigns . Adaptability . The testsets should be easy to update frequently , making them adaptable to improvements in newer systems . Since we automatically extract hard contexts based on MT errors , our testsets are easy to update ; they adapt to errors in newer ( and possibly more accurate ) systems , making the tasks harder over time . We use the system outputs released by WMT and IWSLT for the most recent years ( Nadejde et al. , 2016 ; Bojar et al. , 2017 ; 2018 ; 2019 ; Cettolo et al. , 2016 ; 2017 ) to build our testsets . For De-En , Ru-En and Zh-En , these consist of translation outputs from 68 , 41 and 47 unique systems respectively . Since the data comes from a wide variety of systems , our testsets representatively aggregate different types of errors from several ( arguably SOTA ) models . Also note that the MT models we are benchmarking are not a part of these system submissions to WMT , so there is no potential bias in the testsets . 3.1 ANAPHORA . Anaphora are references to entities that occur elsewhere in a text ; mishandling them can result in ungrammatical sentences or the reader inferring the wrong antecedent , leading to misunderstanding of the text ( Guillou , 2012 ) . We focus specifically on the aspect of incorrect pronoun translations . Testset . To obtain hard contexts for pronoun translation , we look for source texts that lead to erroneous pronoun translations in system outputs . We align the system translations with their references , and collect the cases in which the translated pronouns do not match the reference.4 Our anaphora testset is an updated version of the one proposed by Jwalapuram et al . ( 2019 ) . We filter the system translations based on their list of cases where the translations can be considered wrong , rather than acceptable variants . The corresponding source texts are extracted as a test suite for pronoun translation . This gives us a pronoun benchmark testset of 2564 samples for De-En , 2368 for Ru-En and 1540 for Zh-En . Evaluation . Targeted evaluation of pronouns in MT has been challenging as it is not fair to expect an exact match with the reference . Evaluation methods like APT ( Miculicich Werlen & PopescuBelis , 2017 ) or AutoPRF ( Hardmeier & Federico , 2010 ) are specific to language pairs or lists of pronouns , requiring extensive manual intervention . They have also been criticised for failing to produce evaluations that are consistent with human judgments ( Guillou & Hardmeier , 2018 ) . Jwalapuram et al . ( 2019 ) propose a pairwise ranking model that scores “ good '' pronoun translations ( like in the reference ) higher than “ poor '' pronoun translations ( like in the MT output ) in context , and show that their model is good at making this distinction , along with having high agreements with human judgements . However , they do not rank multiple system translations against each other , which is our main goal ; the absolute scores produced by their model are not useful since it is trained in a pairwise fashion . We devise a way to use their model to score and rank system translations in terms of pronouns . First , we re-train their model with more up-to-date WMT data ( more details in Appendix A.1 ) . We obtain a score for each benchmarked MT system ( S2S , CONCAT , etc . ) translation using the model , plus the corresponding reference sentence . We then normalize the score for each translated sentence by calculating the difference with the reference . To get an overall score for an MT system , the assigned scores are summed across all sentences in the testset . Scoresys = ∑ i ρi ( ref|θ ) − ρi ( sys|θ ) ( 1 ) where ρi ( .|θ ) denotes the score given to sentence i by the pronoun model θ . The systems are ranked based on this overall score , where a lower score indicates a better performance . We conduct a user study to confirm that the model rankings correspond with human judgments , obtaining an agreement of 0.91 between four participants who annotated 100 samples . Appendix A.1 gives details ( e.g. , interface , participants , agreement ) about the study .

Summary: This paper presents a benchmark for discourse phenomena in machine translation. Its main novelty lies in the relatively large scale, spanning three translation directions, four discourse phenomena, and 150-5000 data points per language and phenomenon. A relatively large number of systems from previous work is benchmarked on each test set, and agreement with human judgments is measured.