text: {
    "name" : "1608.04631.pdf",
    "metadata" : {
      "source" : "CRF",
      "title" : "Neural versus Phrase-Based Machine Translation Quality: a Case Study",
      "authors" : [ "Luisa Bentivogli", "Arianna Bisazza", "Mauro Cettolo" ],
      "emails" : [ ],
      "sections" : [ {
        "heading" : "1 Introduction",
        "text" : "The wave of neural models has eventually reached the field of Statistical Machine Translation (SMT). After a period in which Neural MT (NMT) was too computationally costly and resource demanding to compete with state-of-the-art Phrase-Based MT (PBMT)1, the situation changed in 2015. For the first time, in the latest edition of IWSLT2 (Cettolo et\n1We use the generic term phrase-based MT to cover standard phrase-based, hierarchical and syntax-based SMT approaches.\n2International Workshop on Spoken Language Translation (http://workshop2015.iwslt.org/)\nal., 2015), the system described in (Luong and Manning, 2015) overtook a variety of PBMT approaches with a large margin (+5.3 BLEU points) on a difficult language pair like English-German – anticipating what, most likely, will be the new NMT era.\nThis impressive improvement follows the distance reduction previously observed in the WMT 2015 shared translation task (Bojar et al., 2015). Just few months before, the NMT systems described in (Jean et al., 2015b) ranked on par with the best phrase-based models on a couple of language pairs. Such rapid progress stems from the improvement of the recurrent neural network encoderdecoder model, originally proposed in (Sutskever et al., 2014; Cho et al., 2014b), with the use of the attention mechanism (Bahdanau et al., 2015). This evolution has several implications. On one side, NMT represents a simplification with respect to previous paradigms. From a management point of view, similar to PBMT, it allows for a more efficient use of human and data resources with respect to rulebased MT. From the architectural point of view, a large recurrent network trained for end-to-end translation is considerably simpler than traditional MT systems that integrate multiple components and processing steps. On the other side, the NMT process is less transparent than previous paradigms. Indeed, it represents a further step in the evolution from rule-based approaches that explicitly manipulate knowledge, to the statistical/data-driven framework, still comprehensible in its inner workings, to a sub-symbolic framework in which the translation process is totally opaque to the analysis.\nWhat do we know about the strengths of NMT\nar X\niv :1\n60 8.\n04 63\n1v 1\n[ cs\n.C L\n] 1\n6 A\nug 2\nand the weaknesses of PBMT? What are the linguistic phenomena that deep learning translation models can handle with such greater effectiveness? To answer these questions and go beyond poorly informative BLEU scores, we perform the very first comparative analysis of the two paradigms in order to shed light on the factors that differentiate them and determine their large quality differences.\nWe build on evaluation data available for the IWSLT 2015 MT English-German task, and compare the results of the first four top-ranked participants. We choose to focus on one language pair and one task because of the following advantages: (i) three state-of-the art PBMT systems compared against the NMT system on the same data and in the very same period (that of the evaluation campaign); (ii) a challenging language pair in terms of morphology and word order differences; (iii) availability of MT outputs’ post-editing done by professional translators, which is very costly and thus rarely available. In general, post-edits have the advantage of allowing for informative and detailed analyses since they directly point to translation errors. In this specific framework, the high quality data created by professional translators guarantees reliable evaluations. For all these reasons we present our study as a solid contribution to the better understanding of this new paradigm shift in MT.\nAfter reviewing previous work (Section 2), we introduce the analyzed data and the systems that produced them (Section 3). We then present three increasingly fine levels of MT quality analysis. We first investigate how MT systems’ quality varies with specific characteristics of the input, i.e. sentence length and type of content of each talk (Section 4). Then, we focus on differences among MT systems with respect to morphology, lexical, and word order errors (Section 5). Finally, based on the finding that word reordering is the strongest aspect of NMT compared to the other systems, we carry out a finegrained analysis of word order errors (Section 6)."
      }, {
        "heading" : "2 Previous Work",
        "text" : "To date, NMT systems have only been evaluated by BLEU in single-reference setups (Bahdanau et al., 2015; Sutskever et al., 2014; Luong et al., 2015; Jean et al., 2015a; Gülçehre et al., 2015). Ad-\nditionally, the Montreal NMT system submitted to WMT 2015 (Jean et al., 2015b) was part of a manual evaluation experiment where a large number of non-professional annotators were asked to rank the outputs of multiple MT systems (Bojar et al., 2015). Results for the Montreal system were very positive – ranked first in English-German, third in GermanEnglish, English-Czech and Czech-English – which confirmed and strengthened the BLEU results published so far. Unfortunately neither BLEU nor manual ranking judgements tell us which translation aspects are better modeled by different MT frameworks. To this end, a detailed and systematic error analysis of NMT versus PBMT output is required.\nTranslation error analysis, as a way to identify systems’ weaknesses and define priorities for their improvement, has received a fair amount of attention in the MT community. In this work we opt for the automatic detection and classification of translation errors based on manual post-edits of the MT output. We believe this choice provides an optimal trade-off between fully manual error analysis (Farrús Cabeceran et al., 2010; Popović et al., 2013; Daems et al., 2014; Federico et al., 2014; Neubig et al., 2015), which is very costly and complex, and fully automatic error analysis (Popović and Ney, 2011; Irvine et al., 2013), which is noisy and biased towards one or few arbitrary reference translations.\nExisting tools for translation error detection are either based on Word Error Rate (WER) and Position-independent word Error Rate (PER) (Popović, 2011) or on output-reference alignment (Zeman et al., 2011). Regarding error classification, Hjerson (Popović, 2011) detects five main types of word-level errors as defined in (Vilar et al., 2006): morphological, reordering, missing words, extra words, and lexical choice errors. We follow a similar but simpler error classification (morphological, lexical, and word order errors), but detect the errors differently using TER as this is the most natural choice in our evaluation framework based on post-edits (see also Section 3.4). Irvine et al. (2013) propose another word-level error analysis technique specifically focused on lexical choice and aimed at understanding the effects of domain differences on MT. Their error classification is strictly related to model coverage and insensitive to word order differences. The technique requires access to the sys-\ntem’s phrase table and is thus not applicable to NMT, which does not rely on a fixed inventory of sourcetarget translation units extracted from the parallel data.\nPrevious error analyses based on manually postedited translations were presented in (Bojar, 2011; Koponen, 2012; Popović et al., 2013). We are the first to conduct this kind of study on the output of a neural MT system."
      }, {
        "heading" : "3 Experimental Setting",
        "text" : "We perform a number of analyses on data and results of the IWSLT 2015 MT En-De task, which consists in translating manual transcripts of English TED talks into German.\nEvaluation data are publicly available through the WIT3 repository (Cettolo et al., 2012).3"
      }, {
        "heading" : "3.1 Task Data",
        "text" : "TED Talks4 are a collection of rather short speeches (max 18 minutes each, roughly equivalent to 2,500 words) covering a wide variety of topics. All talks have captions, which are translated into many languages by volunteers worldwide. Besides representing a popular benchmark for spoken language technology, TED Talks embed interesting research challenges. Translating TED Talks implies dealing with spoken rather than written language, which is hence expected to be structurally less complex, formal and fluent (Ruiz and Federico, 2014). Moreover, as human translations of the talks are required to follow the structure and rhythm of the English captions, a lower amount of rephrasing and reordering is expected than in ordinary translation of written documents.\nAs regards the English-German language pair, the two languages are interesting since, while belonging to the same language family, they have marked differences in levels of inflection, morphological variation, and word order, especially long-range reordering of verbs."
      }, {
        "heading" : "3.2 Evaluation Data",
        "text" : "Five systems participated in the MT En-De task and were manually evaluated on a representative subset\n3wit3.fbk.eu 4http://www.ted.com/\nof the official 2015 test set. The Human Evaluation (HE) set includes the first half of each of the 12 test talks, for a total of 600 sentences and around 10K words. Five professional translators were asked to post-edit the MT output by applying the minimal edits required to transform it into a fluent sentence with the same meaning as the source sentence. Data were prepared so that all translators equally post-edited the five MT outputs, i.e. 120 sentences for each evaluated system.\nThe resulting evaluation data consist of five new reference translations for each of the sentences in the HE set. Each one of these references represents the targeted translation of the system output from which it was derived, but the other four additional translations can also be used to evaluate each MT system. We will see in the next sections how we exploited the available post-edits in the more suitable way depending on the kind of analysis carried out."
      }, {
        "heading" : "3.3 MT Systems",
        "text" : "Our analysis focuses on the first four top-ranking systems, which include NMT (Luong and Manning, 2015) and three different phrase-based approaches: standard phrase-based (Ha et al., 2015), hierarchical (Jehl et al., 2015) and a combination of phrasebased and syntax-based (Huck and Birch, 2015). Table 1 presents an overview of each system, as well as figures about the training data used.5\nThe phrase+syntax-based (PBSY) system com-\n5Detailed information about training data was kindly made available by participating teams.\nbines the outputs of a string-to-tree decoder, trained with the GHKM algorithm, with those of two standard phrase-based systems featuring, among others, adapted phrase tables and language models enriched with morphological information, hierarchical lexicalized reordering models and different variations of the operational sequence model.\nThe hierarchical phrase-based MT (HPB) system leverages thousands of lexicalised features, datadriven source pre-ordering (dependency tree-based), word-based and class-based LMs, and n-best rescoring models based on syntactic and neural LMs.\nThe standard phrase-based MT (SPB) system features an adapted phrase-table combining in-domain and out-domain data, discriminative word lexicon models, multiple language models (word-, POS- and class-based), data-driven source pre-ordering (POSand constituency syntax-based), n-best re-scoring models based on neural lexicons and neural LMs.\nFinally, the neural MT (NMT) system is a 4-layer long short-term memory (LSTM) network featuring 1,000-dimension word embeddings, attention mechanism, source reversing, 50K source and target vocabularies, and out-of-vocabulary word handling. Training with TED data was performed on top of models trained with large out-domain parallel data.\nWith respect to the use of training data, it is worth noticing that NMT is the only system not employing monolingual data in addition to parallel data. Moreover, NMT and SPB were trained with smaller amounts of parallel data with respect to PBSY and HPB (see Table 1)."
      }, {
        "heading" : "3.4 Translation Edit Rate Measures",
        "text" : "The Translation Edit Rate (TER) (Snover et al., 2006) naturally fits our evaluation framework, where it traces the edits done by post-editors. Also, TER shift operations are reliable indicators of reordering errors, in which we are particularly interested. We exploit the available post-edits in two different ways: (i) for Human-targeted TER (HTER) we compute TER between the machine translation and its manually post-edited version (targeted reference), (ii) for Multi-reference TER (mTER), we compute TER against the closest translation among all available post-edits (i.e. targeted and additional references) for each sentence.\nThroughout sections 4 and 5, we mark a score\nachieved by NMT with the symbol * if this is better than the score of its best competitor at statistical significance level 0.01. Significance tests for HTER and mTER are computed by bootstrap re-sampling, while differences among proportions are assessed via one-tailed z-score tests."
      }, {
        "heading" : "4 Overall Translation Quality",
        "text" : "Table 2 presents overall system results according to HTER and mTER, as well as BLEU computed against the original TED Talks reference translation. We can see that NMT clearly outperforms all other approaches both in terms of BLEU and TER scores. Focusing on mTER results, the gain obtained by NMT over the second best system (PBSY) amounts to 26%. It is also worth noticing that mTER is considerably lower than HTER for each system. This reduction shows that exploiting all the available postedits as references for TER is a viable way to control and overcome post-editors variability, thus ensuring a more reliable and informative evaluation about the real overall performance of MT systems. For this reason, the two following analyses rely on mTER. In particular, we investigate how specific characteristics of input documents affect the system’s overall translation quality, focusing on (i) sentence length and (ii) the different talks composing the dataset."
      }, {
        "heading" : "4.1 Translation quality by sentence length",
        "text" : "Long sentences are known to be difficult to translate by the NMT approach. Following previous work (Cho et al., 2014a; Pouget-Abadie et al., 2014; Bahdanau et al., 2015; Luong et al., 2015), we investigate how sentence length affects overall translation quality. Figure 1 plots mTER scores against source sentence length. NMT clearly outperforms every PBMT system in any length bin, with statistically significant differences. As a general tendency, the\nperformance of all approaches worsens as sentence length increases. However, for sentences longer than 35 words we see that NMT quality degrades more markedly than in PBMT systems. Considering the percentage decrease with respect to the preceding length bin (26-35), we see that the %∆ for NMT (-15.4) is much larger than the average %∆ for the three PBMT systems (-7.9). Hence, this still seems an issue to be addressed for further improving NMT."
      }, {
        "heading" : "4.2 Translation quality by talk",
        "text" : "As we saw in Section 3.1, the TED dataset is very heterogeneous since it consists of talks covering different topics and given by speakers with different styles. It is therefore interesting to evaluate translation quality also at the talk level.\nFigure 2 plots the mTER scores for each of the twelve talks included in the HE set, sorted in ascending order of NMT scores. In all talks, the NMT system outperforms the PBMT systems in a statistically significant way.\nWe analysed different factors which could impact translation quality in order to understand if they correlate with such performance differences. We studied three features which are typically considered as indicators of complexity (see (François and Fairon, 2012) for an overview), namely (i) the length of the talk, (ii) its average sentence length, and (iii) the type-token ratio6 (TTR) which – measuring lexical\n6The type-token-ratio of a text is calculated dividing the number of word types (vocabulary) by the total number of word tokens (occurrences).\ndiversity – reflects the size of a speaker’s vocabulary and the variety of subject matter in a text.\nFor the first two features we did not find any correlation; on the contrary, we found a moderate Pearson correlation (R=0.7332) between TTR and the mTER gains of NMT over its closest competitor in each talk. This result suggests that NMT is able to cope with lexical diversity better than any other considered approach."
      }, {
        "heading" : "5 Analysis of Translation Errors",
        "text" : "We now turn to analyze which types of linguistic errors characterize NMT vs. PBMT. In the literature, various error taxonomies covering different levels of granularity have been developed (Flanagan, 1994; Vilar et al., 2006; Farrús Cabeceran et al., 2010; Stymne and Ahrenberg, 2012; Lommel et al., 2014). We focus on three error categories, namely (i) morphology errors, (ii) lexical errors, and (iii) word order errors. As for lexical errors, a number of existing taxonomies further distinguish among translation errors due to missing words, extra words, or wrong lexical choice. However, given the proved difficulty of disambiguating between these three subclasses (Popović and Ney, 2011; Fishel et al., 2012), we prefer to rely on a more coarse-grained linguistic error classification where lexical errors include all of them (Farrús Cabeceran et al., 2010).\nFor error analysis we rely on HTER results under the assumption that, since the targeted translation is generated by post-editing the given MT output, this method is particularly informative to spot MT errors.\nWe are aware that translators’ subjectivity is still an issue (see Section 4), however in this more finegrained analysis we prefer to focus on what a human implicitly annotated as a translation error. This particularly holds in our specific evaluation framework, where the goal is not to measure the absolute number of errors made by each system, but to compare systems among each other. Moreover, the postedits collected for each MT output within IWSLT allow for a fair and reliable comparison since systems were equally post-edited by all translators (see Section 3.2), making all analyses uniformly affected by such variability."
      }, {
        "heading" : "5.1 Morphology errors",
        "text" : "A morphology error occurs when a generated word form is wrong but its corresponding base form (lemma) is correct. Thus, we assess the ability of systems to deal with morphology by comparing the HTER score computed on the surface forms (i.e. morphologically inflected words) with the HTER score obtained on the corresponding lemmas. The additional matches counted on lemmas with respect to word forms indicate morphology errors. Thus, the closer the two HTER scores, the more accurate the system in handling morphology.\nTo carry out this analysis, the lemmatized (and POS tagged) version of both MT outputs and corresponding post-edits was produced with the German parser ParZu (Sennrich et al., 2013). Then, the HTER-based evaluation was slightly adapted in order to be better suited to an accurate detection of morphology errors. First, punctuation was removed since – not being subject to morphological inflection – it could smooth the results. Second, shift errors were not considered. A word form or a lemma that matches a corresponding word or lemma in the postedit, but is in the wrong position with respect to it, is counted as a shift error in TER. Instead – when focusing on morphology – exact matches are not errors, regardless their position in the text.7\nTable 3 presents HTER scores on word forms and lemmas, as well as their percentage difference which gives an indication of morphology errors. We can see that NMT generates translations which are mor-\n7Note that the TER score calculated by setting to 0 the cost of shifts approximates the Position-independent Error Rate (Tillmann et al., 1997).\nphologically more correct than the other systems. In particular, the %∆ for NMT (-13.7) is lower than that of the second best system (PBSY, -16.9) by 3.2% absolute points, leading to a percentage gain of around 19%. We can thus say that NMT makes at least 19% less morphology errors than any other PBMT system."
      }, {
        "heading" : "5.2 Lexical errors",
        "text" : "Another important feature of MT systems is their ability to choose lexically appropriate words. In order to compare systems under this aspect, we consider HTER results at the lemma level as a way to abstract from morphology errors and focus only on actual lexical choice problems. The evaluation on the lemmatised version of the data performed to identify morphology errors fits to this purpose, since its driving assumptions (i.e. punctuation can be excluded and lemmas in the wrong order are not errors) hold for lexical errors too.\nThe lemma column of Table 3 shows that NMT outperforms the other systems. More precisely, the NMT score (18.7) is better than the second best (PBSY, 22.5) by 3.8% absolute points. This corresponds to a relative gain of about 17%, meaning that NMT makes at least 17% less lexical errors than any PBMT system. Similarly to what observed for morphology errors, this can be considered a remarkable improvement over the state of the art."
      }, {
        "heading" : "5.3 Word order errors",
        "text" : "To analyse reordering errors, we start by focusing on shift operations identified by the HTER metrics. The first three columns of Table 4 show, respectively: (i) the number of words generated by each system (ii) the number of shifts required to align each system output to the corresponding post-edit; and (iii) the corresponding percentage of shift errors. Notice that the shift error percentages are incorporated in\nthe HTER scores reported in Table 2. We can see in the table that the percentage of shift errors in NMT translations is definitely lower than for the other systems. The gap between NMT and the second best system (PBSY) is about 50%.\nIt should be recalled that these numbers only refer to shifts detected by HTER, that is (groups of) words of the MT output and post-edits that are identical but occurring in different positions. Words that had to be moved and modified at the same time (for instance replaced by a synonym or a morphological variant) are not counted in HTER shift figures, but are detected as substitution, insertion or deletion operations. To ensure that our reordering evaluation is not biased towards the alignment between the MT output and the post-edit performed by HTER, we run an additional assessment using KRS, or Kendall Reordering Score (Birch et al., 2010), which measures the similarity between the source-reference reorderings and the source-MT output reorderings.8 Being based on bilingual word alignment via the source sentence, KRS detects reordering errors also when post-edit and MT words are not identical. Also unlike TER, KRS is sensitive to the distance between the position of a word in the MT output and that in the reference.\nLooking at the last column of Table 4, we can say that our observations on HTER are confirmed by the KRS results: the reorderings performed by NMT are much more accurate than those performed by any PBMT system.9 Moreover, according to the approximate randomization test, KRS differences are statis-\n8To compute the word alignments required by KRS, we used the FastAlign tool (Dyer et al., 2013).\n9To put our results into perspective, note that Birch (2011) reports a difference of 5 KRS points between the translations of a PBMT system and those produced by four human translators tested against each other, in a Chinese-English experiment.\ntically significant between NMT and all other systems, but not among the three PBMT systems.\nGiven the concordant results of our two quantitative analyses, we conclude that one of the major strengths of the NMT approach is its ability to place German words in the right position even when this requires considerable reordering. This outcome calls for a deeper investigation, which is carried out in the following section."
      }, {
        "heading" : "6 Fine-grained Word Order Error Analysis",
        "text" : "We have observed that word reordering is a very strong aspect of NMT compared to PBMT, according to both HTER and KRS. To better understand this finding, we investigate whether reordering errors concentrate on specific linguistic constructions across our systems. Using the POS tagging and dependency parsing of the post-edits produced by ParZu, we classify the shift operations detected by HTER and count how often a word with a given POS label was misplaced by each of the systems (alone or as part of a shifted block). For each word class, we also compute the percentage order error reduction of NMT with respect to the PBMT system that has highest reordering accuracy overall, that is PBSY. Results are presented in Table 5, ranked by NMT-vsPBSY gain. We omit punctuation as well as word classes that were shifted less than 10 times by all systems. Examples of salient word order error types are presented in Table 6.\nThe upper part of Table 5 shows that verbs are by far the most often misplaced word category in all PBMT systems – an issue already known to affect standard phrase-based SMT between German and English (Bisazza and Federico, 2013). Reordering is particularly difficult when translating into German, since the position of verbs in this language varies according to the clause type (e.g. main versus subordinate). Our results show that even syntaxinformed PBMT does not solve this issue. Using syntax at decoding time, as done by one of the systems combined within PBSY, appears to be a better strategy than using it for source pre-ordering, as done by the HPB and SPB systems. However this only results in a moderate reduction of verb reordering errors (-12% and -25% versus HPB and SPB\nrespectively). On the contrary, NMT reduces verb order errors by an impressive -70% with respect to PBSY (-74% and -77% versus HPB and SPB respectively) despite being trained on raw parallel data without any syntactic annotation, nor explicit modeling of word reordering. This result shows that the recurrent neural language model at the core of the NMT architecture is very successful at generating well-formed sentences even in languages with less predictable word order, like German (see examples in Table 6(a,b)). NMT, though, gains notably less on nouns (-47%), which is the second most often misplaced word category in PBSY. More insight on this is provided by the lower part of the table, where reordering errors are divided by their dependency label as well as POS tag. Here we see that order errors on nouns are notably reduced by NMT when they act as syntactic objects (-65% obja:N) but less when they act as preposition complements (-36% pn:N) or subjects (-33% subj:N).\nThe smallest NMT-vs-PBSY gains are observed on prepositions (-18% pp:PREP), negation particles\n(-17% PTKNEG) and articles (-4% det:ART). Manual inspection of a data sample reveals that misplaced prepositions are often part of misplaced prepositional phrases acting, for instance, as temporal or instrumental adjuncts (e.g. ‘in my life’, ‘with this video’). In these cases, the original MT output is overall understandable and grammatical, but does not conform to the order of German semantic arguments that is consistently preferred by post-editors (see example in Table 6(c)). Articles, due to their commonness, are often misaligned by HTER and marked as shift errors instead of being marked as two unrelated substitutions. Finally, negation particles account for less than 1% of the target tokens but play a key role in determining the sentence meaning. Looking closely at some error examples, we found that the correct placement of the German particle nicht was determined by the focus of negation in the source sentence, which is difficult to detect in English. For instance in Table 6(d) two interpretations are possible (‘that did not work’ or ‘that worked, but not for systematic reasons’), each resulting in a dif-\nferent, but equally grammatical, location of nicht. In fact, negation-focus detection calls for a deep understanding of the sentence semantics, often requiring extra-sentential context (Blanco and Moldovan, 2011). When faced with this kind of translation decisions, NMT performs as bad as its competitors.\nIn summary, our fine-grained analysis confirms that NMT concentrates its word order improvements on important linguistic constituents and, specifically in English-German, is very close to solving the infamous problem of long-range verb reordering which so many PBMT approaches have only poorly managed to handle. On the other hand, NMT still struggles with more subtle translation decisions depending, for instance, on the semantic ordering of adjunct prepositional phrases or on the focus of negation."
      }, {
        "heading" : "7 Conclusions",
        "text" : "We have analysed the output of four state-of-theart MT systems that participated in the English-toGerman task of the IWSLT 2015 evaluation campaign. Our selected runs were produced by three phrase-based MT systems and a neural MT system. The analysis leveraged high quality post-edits of the MT outputs, which allowed us to profile systems with respect to reliable measures of post-editing effort and translation error types.\nThe outcomes of the analysis confirm that NMT has significantly pushed ahead the state of the art, especially in a language pair involving rich morphology prediction and significant word reordering. To summarize our findings: (i) NMT generates outputs that considerably lower the overall post-edit effort with respect to the best PBMT system (-26%); (ii) NMT outperforms PBMT systems on all sentence lengths, although its performance degrades faster with the input length than its competitors; (iii) NMT seems to have an edge especially on lexically rich texts; (iv) NMT output contains less morphology errors (-19%), less lexical errors (-17%), and substantially less word order errors (-50%) than its closest competitor for each error type; (v) concerning word order, NMT shows an impressive improvement in the placement of verbs (-70% errors).\nWhile NMT has proved superior to PBMT with respect to all error types that we have investigated, our analysis has also pointed out some aspects of NMT that deserve further work, such as the handling of long sentences and the reordering of particular linguistic constituents requiring a deep semantic understanding of text. Machine translation is definitely not a solved problem, but the time is finally ripe to tackle its most intricate aspects."
      }, {
        "heading" : "Acknowledgments",
        "text" : "FBK authors were supported by the CRACKER and QT21 projects, which received funding from the European Unions Horizon 2020 research and innovation programme under grant no. 645357 and no. 645452, respectively. AB’s work was funded in part by the Netherlands Organisation for Scientific Research (NWO) under project numbers 639.022.213 and 612.001.218."
      } ],
      "references" : [ {
        "title" : "Neural machine translation by jointly learning to align and translate",
        "author" : [ "Kyunghyun Cho", "Yoshua Bengio" ],
        "venue" : "In Proc. of ICLR,",
        "citeRegEx" : "Bahdanau et al\\.,? \\Q2015\\E",
        "shortCiteRegEx" : "Bahdanau et al\\.",
        "year" : 2015
      }, {
        "title" : "Metrics for MT evaluation: evaluating reordering",
        "author" : [ "Miles Osborne", "Phil Blunsom" ],
        "venue" : "Machine Translation,",
        "citeRegEx" : "Birch et al\\.,? \\Q2010\\E",
        "shortCiteRegEx" : "Birch et al\\.",
        "year" : 2010
      }, {
        "title" : "Reordering Metrics for Statistical Machine Translation",
        "author" : [ "Alexandra Birch" ],
        "venue" : "Ph.D. thesis, School of Informatics,",
        "citeRegEx" : "Birch.,? \\Q2011\\E",
        "shortCiteRegEx" : "Birch.",
        "year" : 2011
      }, {
        "title" : "Efficient solutions for word reordering in German-English phrase-based statistical machine translation",
        "author" : [ "Bisazza", "Federico2013] Arianna Bisazza", "Marcello Federico" ],
        "venue" : "In Proc. of WMT, Sofia, Bulgaria",
        "citeRegEx" : "Bisazza et al\\.,? \\Q2013\\E",
        "shortCiteRegEx" : "Bisazza et al\\.",
        "year" : 2013
      }, {
        "title" : "Semantic representation of negation using focus detection",
        "author" : [ "Blanco", "Moldovan2011] Eduardo Blanco", "Dan Moldovan" ],
        "venue" : "In Proc. of ACL-HLT,",
        "citeRegEx" : "Blanco et al\\.,? \\Q2011\\E",
        "shortCiteRegEx" : "Blanco et al\\.",
        "year" : 2011
      }, {
        "title" : "Analyzing error types in English-Czech machine translation",
        "author" : [ "Ondrej Bojar" ],
        "venue" : "The Prague Bulletin of Mathematical Linguistic,",
        "citeRegEx" : "Bojar.,? \\Q2011\\E",
        "shortCiteRegEx" : "Bojar.",
        "year" : 2011
      }, {
        "title" : "WIT: Web Inventory of Transcribed and Translated Talks",
        "author" : [ "Christian Girardi", "Marcello Federico" ],
        "venue" : "In Proc. of EAMT,",
        "citeRegEx" : "Cettolo et al\\.,? \\Q2012\\E",
        "shortCiteRegEx" : "Cettolo et al\\.",
        "year" : 2012
      }, {
        "title" : "The IWSLT 2015 evaluation campaign",
        "author" : [ "Jan Niehues", "Sebastian Stüker", "Luisa Bentivogli", "Roldano Cattoni", "Marcello Federico" ],
        "venue" : "In Proc. of IWSLT, Da Nang, Vietnam",
        "citeRegEx" : "Cettolo et al\\.,? \\Q2015\\E",
        "shortCiteRegEx" : "Cettolo et al\\.",
        "year" : 2015
      }, {
        "title" : "On the properties of neural machine translation: encoder– decoder approaches",
        "author" : [ "Cho et al.2014a] Kyunghyun Cho", "Bart van Merriënboer", "Dzmitry Bahdanau", "Yoshua Bengio" ],
        "venue" : "In Proc. of SSST-8,",
        "citeRegEx" : "Cho et al\\.,? \\Q2014\\E",
        "shortCiteRegEx" : "Cho et al\\.",
        "year" : 2014
      }, {
        "title" : "Learning phrase representations using RNN encoder–decoder for statistical machine translation",
        "author" : [ "Cho et al.2014b] Kyunghyun Cho", "Bart van Merrienboer", "Caglar Gulcehre", "Dzmitry Bahdanau", "Fethi Bougares", "Holger Schwenk", "Yoshua Bengio" ],
        "venue" : null,
        "citeRegEx" : "Cho et al\\.,? \\Q2014\\E",
        "shortCiteRegEx" : "Cho et al\\.",
        "year" : 2014
      }, {
        "title" : "On the origin of errors: a finegrained analysis of MT and PE errors and their relationship",
        "author" : [ "Daems et al.2014] Joke Daems", "Lieve Macken", "Sonia Vandepitte" ],
        "venue" : "In Proc. of LREC,",
        "citeRegEx" : "Daems et al\\.,? \\Q2014\\E",
        "shortCiteRegEx" : "Daems et al\\.",
        "year" : 2014
      }, {
        "title" : "A simple, fast, and effective reparameterization of IBM model 2",
        "author" : [ "Dyer et al.2013] Chris Dyer", "Victor Chahuneau", "Noah A. Smith" ],
        "venue" : "In Proc. of NACL-HLT,",
        "citeRegEx" : "Dyer et al\\.,? \\Q2013\\E",
        "shortCiteRegEx" : "Dyer et al\\.",
        "year" : 2013
      }, {
        "title" : "Linguistic-based evaluation criteria to identify statistical machine translation errors",
        "author" : [ "Marta Ruiz Costa-Jussà", "José Bernardo Mariño Acebal", "José Adrián Rodrı́guez Fonollosa" ],
        "venue" : "In Proc. of EAMT,",
        "citeRegEx" : "Cabeceran et al\\.,? \\Q2010\\E",
        "shortCiteRegEx" : "Cabeceran et al\\.",
        "year" : 2010
      }, {
        "title" : "Assessing the impact of translation errors on machine translation quality with mixed-effects models",
        "author" : [ "Matteo Negri", "Luisa Bentivogli", "Marco Turchi" ],
        "venue" : "In Proc. of EMNLP,",
        "citeRegEx" : "Federico et al\\.,? \\Q2014\\E",
        "shortCiteRegEx" : "Federico et al\\.",
        "year" : 2014
      }, {
        "title" : "Terra: a collection of translation error-annotated corpora",
        "author" : [ "Fishel et al.2012] Mark Fishel", "Ondrej Bojar", "Maja Popović" ],
        "venue" : "In Proc. of LREC,",
        "citeRegEx" : "Fishel et al\\.,? \\Q2012\\E",
        "shortCiteRegEx" : "Fishel et al\\.",
        "year" : 2012
      }, {
        "title" : "Error classification for MT evaluation",
        "author" : [ "Mary Flanagan" ],
        "venue" : "In Proc. of AMTA,",
        "citeRegEx" : "Flanagan.,? \\Q1994\\E",
        "shortCiteRegEx" : "Flanagan.",
        "year" : 1994
      }, {
        "title" : "An “AI readability” formula for French as a foreign language",
        "author" : [ "François", "Fairon2012] Thomas François", "Cédrick Fairon" ],
        "venue" : "In Proc. of EMNLP-CoNLL,",
        "citeRegEx" : "François et al\\.,? \\Q2012\\E",
        "shortCiteRegEx" : "François et al\\.",
        "year" : 2012
      }, {
        "title" : "On using monolingual corpora in neural machine translation. CoRR, abs/1503.03535",
        "author" : [ "Orhan Firat", "Kelvin Xu", "Kyunghyun Cho", "Loı̈c Barrault", "Huei-Chi Lin", "Fethi Bougares", "Holger Schwenk", "Yoshua Bengio" ],
        "venue" : null,
        "citeRegEx" : "Gülçehre et al\\.,? \\Q2015\\E",
        "shortCiteRegEx" : "Gülçehre et al\\.",
        "year" : 2015
      }, {
        "title" : "The KIT translation systems for IWSLT",
        "author" : [ "Ha et al.2015] Thanh-Le Ha", "Jan Niehues", "Eunah Cho", "Mohammed Mediani", "Alex Waibel" ],
        "venue" : "In Proc. of IWSLT, Da Nang, Vietnam",
        "citeRegEx" : "Ha et al\\.,? \\Q2015\\E",
        "shortCiteRegEx" : "Ha et al\\.",
        "year" : 2015
      }, {
        "title" : "The Edinburgh machine translation systems for IWSLT",
        "author" : [ "Huck", "Birch2015] Matthias Huck", "Alexandra Birch" ],
        "venue" : "In Proc. of IWSLT, Da Nang, Vietnam",
        "citeRegEx" : "Huck et al\\.,? \\Q2015\\E",
        "shortCiteRegEx" : "Huck et al\\.",
        "year" : 2015
      }, {
        "title" : "Measuring machine translation errors in new domains. Transactions of the Association for Computational Linguistics, 1:429–440",
        "author" : [ "Irvine et al.2013] Ann Irvine", "John Morgan", "Marine Carpuat", "Hal Daumé III", "Dragos Munteanu" ],
        "venue" : null,
        "citeRegEx" : "Irvine et al\\.,? \\Q2013\\E",
        "shortCiteRegEx" : "Irvine et al\\.",
        "year" : 2013
      }, {
        "title" : "On using very large target vocabulary for neural machine translation",
        "author" : [ "Jean et al.2015a] Sébastien Jean", "Kyunghyun Cho", "Roland Memisevic", "Yoshua Bengio" ],
        "venue" : "In Proc. of ACL-IJCNLP,",
        "citeRegEx" : "Jean et al\\.,? \\Q2015\\E",
        "shortCiteRegEx" : "Jean et al\\.",
        "year" : 2015
      }, {
        "title" : "2015b. Montreal neural machine translation systems for WMT15",
        "author" : [ "Jean et al.2015b] Sébastien Jean", "Orhan Firat", "Kyunghyun Cho", "Roland Memisevic", "Yoshua Bengio" ],
        "venue" : "In Proc. of WMT,",
        "citeRegEx" : "Jean et al\\.,? \\Q2015\\E",
        "shortCiteRegEx" : "Jean et al\\.",
        "year" : 2015
      }, {
        "title" : "The Heidelberg university English-German translation system for IWSLT",
        "author" : [ "Jehl et al.2015] Laura Jehl", "Patrick Simianer", "Julian Hitschler", "Stefan Riezler" ],
        "venue" : "In Proc. of IWSLT, Da Nang, Vietnam",
        "citeRegEx" : "Jehl et al\\.,? \\Q2015\\E",
        "shortCiteRegEx" : "Jehl et al\\.",
        "year" : 2015
      }, {
        "title" : "Comparing human perceptions of post-editing effort with postediting operations",
        "author" : [ "Maarit Koponen" ],
        "venue" : "In Proc. of WMT,",
        "citeRegEx" : "Koponen.,? \\Q2012\\E",
        "shortCiteRegEx" : "Koponen.",
        "year" : 2012
      }, {
        "title" : "Using a new analytic measure for the annotation and analysis of MT errors on real data",
        "author" : [ "Lommel et al.2014] Arle Lommel", "Aljoscha Burchardt", "Maja Popović", "Kim Harris", "Eleftherios Avramidis", "Hans Uszkoreit" ],
        "venue" : "In Proc. of EAMT,",
        "citeRegEx" : "Lommel et al\\.,? \\Q2014\\E",
        "shortCiteRegEx" : "Lommel et al\\.",
        "year" : 2014
      }, {
        "title" : "Stanford neural machine translation systems for spoken language domains",
        "author" : [ "Luong", "Manning2015] Minh-Thang Luong", "Christopher D Manning" ],
        "venue" : "In Proc. of IWSLT, Da Nang, Vietnam",
        "citeRegEx" : "Luong et al\\.,? \\Q2015\\E",
        "shortCiteRegEx" : "Luong et al\\.",
        "year" : 2015
      }, {
        "title" : "Effective approaches to attention-based neural machine translation",
        "author" : [ "Luong et al.2015] Thang Luong", "Hieu Pham", "Christopher D. Manning" ],
        "venue" : "In Proc. of EMNLP,",
        "citeRegEx" : "Luong et al\\.,? \\Q2015\\E",
        "shortCiteRegEx" : "Luong et al\\.",
        "year" : 2015
      }, {
        "title" : "Neural Reranking Improves Subjective Quality of Machine Translation: NAIST at WAT2015",
        "author" : [ "Neubig et al.2015] Graham Neubig", "Makoto Morishita", "Satoshi Nakamura" ],
        "venue" : "In Proc. of WAT2015,",
        "citeRegEx" : "Neubig et al\\.,? \\Q2015\\E",
        "shortCiteRegEx" : "Neubig et al\\.",
        "year" : 2015
      }, {
        "title" : "Towards automatic error analysis of machine translation output",
        "author" : [ "Popović", "Ney2011] Maja Popović", "Hermann Ney" ],
        "venue" : "Computational Linguistics,",
        "citeRegEx" : "Popović et al\\.,? \\Q2011\\E",
        "shortCiteRegEx" : "Popović et al\\.",
        "year" : 2011
      }, {
        "title" : "Learning from human judgments of machine translation output",
        "author" : [ "Popović et al.2013] Maja Popović", "Eleftherios Avramidis", "Aljoscha Burchardt", "Sabine Hunsicker", "Sven Schmeier", "Cindy Tscherwinka", "David Vilar", "Hans Uszkoreit" ],
        "venue" : null,
        "citeRegEx" : "Popović et al\\.,? \\Q2013\\E",
        "shortCiteRegEx" : "Popović et al\\.",
        "year" : 2013
      }, {
        "title" : "Hjerson: an open source tool for automatic error classification of machine translation output",
        "author" : [ "Maja Popović" ],
        "venue" : "The Prague Bulletin of Mathematical Linguistic,",
        "citeRegEx" : "Popović.,? \\Q2011\\E",
        "shortCiteRegEx" : "Popović.",
        "year" : 2011
      }, {
        "title" : "Overcoming the curse of sentence length for neural machine translation using automatic segmentation",
        "author" : [ "Dzmitry Bahdanau", "Bart van Merrienboer", "Kyunghyun Cho", "Yoshua Bengio" ],
        "venue" : "In Proc. of SSST-8, Doha,",
        "citeRegEx" : "Pouget.Abadie et al\\.,? \\Q2014\\E",
        "shortCiteRegEx" : "Pouget.Abadie et al\\.",
        "year" : 2014
      }, {
        "title" : "Complexity of spoken versus written language for machine translation",
        "author" : [ "Ruiz", "Federico2014] Nicholas Ruiz", "Marcello Federico" ],
        "venue" : "In Proc. of EAMT,",
        "citeRegEx" : "Ruiz et al\\.,? \\Q2014\\E",
        "shortCiteRegEx" : "Ruiz et al\\.",
        "year" : 2014
      }, {
        "title" : "Exploiting synergies between open resources for German dependency parsing",
        "author" : [ "Martin Volk", "Gerold Schneider" ],
        "venue" : null,
        "citeRegEx" : "Sennrich et al\\.,? \\Q2013\\E",
        "shortCiteRegEx" : "Sennrich et al\\.",
        "year" : 2013
      }, {
        "title" : "A study of translation edit rate with targeted human annotation",
        "author" : [ "Bonnie Dorr", "Rich Schwartz", "Linnea Micciulla", "John Makhoul" ],
        "venue" : "In Proc. of AMTA,",
        "citeRegEx" : "Snover et al\\.,? \\Q2006\\E",
        "shortCiteRegEx" : "Snover et al\\.",
        "year" : 2006
      }, {
        "title" : "On the practice of error analysis for machine translation evaluation",
        "author" : [ "Stymne", "Ahrenberg2012] Sara Stymne", "Lars Ahrenberg" ],
        "venue" : "In Proc. of LREC,",
        "citeRegEx" : "Stymne et al\\.,? \\Q2012\\E",
        "shortCiteRegEx" : "Stymne et al\\.",
        "year" : 2012
      }, {
        "title" : "Sequence to sequence learning with neural networks",
        "author" : [ "Oriol Vinyals", "Quoc V. Le" ],
        "venue" : "In Proc. of NIPS,",
        "citeRegEx" : "Sutskever et al\\.,? \\Q2014\\E",
        "shortCiteRegEx" : "Sutskever et al\\.",
        "year" : 2014
      }, {
        "title" : "Accelerated DP based search for statistical translation",
        "author" : [ "Stephan Vogel", "Hermann Ney", "Alexander Zubiaga", "Hassan Sawaf" ],
        "venue" : "In Proc. of Eurospeech,",
        "citeRegEx" : "Tillmann et al\\.,? \\Q1997\\E",
        "shortCiteRegEx" : "Tillmann et al\\.",
        "year" : 1997
      }, {
        "title" : "Error analysis of statistical machine translation output",
        "author" : [ "Vilar et al.2006] David Vilar", "Jia Xu", "Luis Fernando d’Haro", "Hermann Ney" ],
        "venue" : "In Proc. of LREC,",
        "citeRegEx" : "Vilar et al\\.,? \\Q2006\\E",
        "shortCiteRegEx" : "Vilar et al\\.",
        "year" : 2006
      }, {
        "title" : "Addicter: what is wrong with my translations",
        "author" : [ "Zeman et al.2011] Daniel Zeman", "Mark Fishel", "Jan Berka", "Ondrej Bojar" ],
        "venue" : "The Prague Bulletin of Mathematical Linguistic,",
        "citeRegEx" : "Zeman et al\\.,? \\Q2011\\E",
        "shortCiteRegEx" : "Zeman et al\\.",
        "year" : 2011
      } ],
      "referenceMentions" : [ {
        "referenceID" : 37,
        "context" : "Such rapid progress stems from the improvement of the recurrent neural network encoderdecoder model, originally proposed in (Sutskever et al., 2014; Cho et al., 2014b), with the use of the at-",
        "startOffset" : 124,
        "endOffset" : 167
      }, {
        "referenceID" : 0,
        "context" : "tention mechanism (Bahdanau et al., 2015).",
        "startOffset" : 18,
        "endOffset" : 41
      }, {
        "referenceID" : 0,
        "context" : "To date, NMT systems have only been evaluated by BLEU in single-reference setups (Bahdanau et al., 2015; Sutskever et al., 2014; Luong et al., 2015; Jean et al., 2015a; Gülçehre et al., 2015).",
        "startOffset" : 81,
        "endOffset" : 191
      }, {
        "referenceID" : 37,
        "context" : "To date, NMT systems have only been evaluated by BLEU in single-reference setups (Bahdanau et al., 2015; Sutskever et al., 2014; Luong et al., 2015; Jean et al., 2015a; Gülçehre et al., 2015).",
        "startOffset" : 81,
        "endOffset" : 191
      }, {
        "referenceID" : 26,
        "context" : "To date, NMT systems have only been evaluated by BLEU in single-reference setups (Bahdanau et al., 2015; Sutskever et al., 2014; Luong et al., 2015; Jean et al., 2015a; Gülçehre et al., 2015).",
        "startOffset" : 81,
        "endOffset" : 191
      }, {
        "referenceID" : 17,
        "context" : "To date, NMT systems have only been evaluated by BLEU in single-reference setups (Bahdanau et al., 2015; Sutskever et al., 2014; Luong et al., 2015; Jean et al., 2015a; Gülçehre et al., 2015).",
        "startOffset" : 81,
        "endOffset" : 191
      }, {
        "referenceID" : 20,
        "context" : ", 2015), which is very costly and complex, and fully automatic error analysis (Popović and Ney, 2011; Irvine et al., 2013), which is noisy and biased towards one or few arbitrary reference translations.",
        "startOffset" : 78,
        "endOffset" : 122
      }, {
        "referenceID" : 31,
        "context" : "(Popović, 2011) or on output-reference alignment (Zeman et al.",
        "startOffset" : 0,
        "endOffset" : 15
      }, {
        "referenceID" : 40,
        "context" : "(Popović, 2011) or on output-reference alignment (Zeman et al., 2011).",
        "startOffset" : 49,
        "endOffset" : 69
      }, {
        "referenceID" : 31,
        "context" : "Regarding error classification, Hjerson (Popović, 2011) detects five main types of word-level errors as defined in (Vilar et al.",
        "startOffset" : 40,
        "endOffset" : 55
      }, {
        "referenceID" : 39,
        "context" : "Regarding error classification, Hjerson (Popović, 2011) detects five main types of word-level errors as defined in (Vilar et al., 2006): morphological, reordering, missing words, extra words, and lexical choice errors.",
        "startOffset" : 115,
        "endOffset" : 135
      }, {
        "referenceID" : 20,
        "context" : "Irvine et al. (2013) propose another word-level error analysis technique specifically focused on lexical choice and aimed at understanding the effects of domain differences on MT.",
        "startOffset" : 0,
        "endOffset" : 21
      }, {
        "referenceID" : 5,
        "context" : "Previous error analyses based on manually postedited translations were presented in (Bojar, 2011; Koponen, 2012; Popović et al., 2013).",
        "startOffset" : 84,
        "endOffset" : 134
      }, {
        "referenceID" : 24,
        "context" : "Previous error analyses based on manually postedited translations were presented in (Bojar, 2011; Koponen, 2012; Popović et al., 2013).",
        "startOffset" : 84,
        "endOffset" : 134
      }, {
        "referenceID" : 30,
        "context" : "Previous error analyses based on manually postedited translations were presented in (Bojar, 2011; Koponen, 2012; Popović et al., 2013).",
        "startOffset" : 84,
        "endOffset" : 134
      }, {
        "referenceID" : 6,
        "context" : "Evaluation data are publicly available through the WIT3 repository (Cettolo et al., 2012).",
        "startOffset" : 67,
        "endOffset" : 89
      }, {
        "referenceID" : 18,
        "context" : "Our analysis focuses on the first four top-ranking systems, which include NMT (Luong and Manning, 2015) and three different phrase-based approaches: standard phrase-based (Ha et al., 2015), hierarchical (Jehl et al.",
        "startOffset" : 171,
        "endOffset" : 188
      }, {
        "referenceID" : 23,
        "context" : ", 2015), hierarchical (Jehl et al., 2015) and a combination of phrasebased and syntax-based (Huck and Birch, 2015).",
        "startOffset" : 22,
        "endOffset" : 41
      }, {
        "referenceID" : 35,
        "context" : "The Translation Edit Rate (TER) (Snover et al., 2006) naturally fits our evaluation framework, where it traces the edits done by post-editors.",
        "startOffset" : 32,
        "endOffset" : 53
      }, {
        "referenceID" : 32,
        "context" : "Following previous work (Cho et al., 2014a; Pouget-Abadie et al., 2014; Bahdanau et al., 2015; Luong et al., 2015), we investigate how sentence length affects overall translation quality.",
        "startOffset" : 24,
        "endOffset" : 114
      }, {
        "referenceID" : 0,
        "context" : "Following previous work (Cho et al., 2014a; Pouget-Abadie et al., 2014; Bahdanau et al., 2015; Luong et al., 2015), we investigate how sentence length affects overall translation quality.",
        "startOffset" : 24,
        "endOffset" : 114
      }, {
        "referenceID" : 26,
        "context" : "Following previous work (Cho et al., 2014a; Pouget-Abadie et al., 2014; Bahdanau et al., 2015; Luong et al., 2015), we investigate how sentence length affects overall translation quality.",
        "startOffset" : 24,
        "endOffset" : 114
      }, {
        "referenceID" : 15,
        "context" : "granularity have been developed (Flanagan, 1994; Vilar et al., 2006; Farrús Cabeceran et al., 2010; Stymne and Ahrenberg, 2012; Lommel et al., 2014).",
        "startOffset" : 32,
        "endOffset" : 148
      }, {
        "referenceID" : 39,
        "context" : "granularity have been developed (Flanagan, 1994; Vilar et al., 2006; Farrús Cabeceran et al., 2010; Stymne and Ahrenberg, 2012; Lommel et al., 2014).",
        "startOffset" : 32,
        "endOffset" : 148
      }, {
        "referenceID" : 25,
        "context" : "granularity have been developed (Flanagan, 1994; Vilar et al., 2006; Farrús Cabeceran et al., 2010; Stymne and Ahrenberg, 2012; Lommel et al., 2014).",
        "startOffset" : 32,
        "endOffset" : 148
      }, {
        "referenceID" : 14,
        "context" : "However, given the proved difficulty of disambiguating between these three subclasses (Popović and Ney, 2011; Fishel et al., 2012), we prefer to rely on a more coarse-grained linguistic error classification where lexical errors include all of them (Farrús Cabeceran et al.",
        "startOffset" : 86,
        "endOffset" : 130
      }, {
        "referenceID" : 34,
        "context" : "responding post-edits was produced with the German parser ParZu (Sennrich et al., 2013).",
        "startOffset" : 64,
        "endOffset" : 87
      }, {
        "referenceID" : 38,
        "context" : "Note that the TER score calculated by setting to 0 the cost of shifts approximates the Position-independent Error Rate (Tillmann et al., 1997).",
        "startOffset" : 119,
        "endOffset" : 142
      }, {
        "referenceID" : 1,
        "context" : "output and the post-edit performed by HTER, we run an additional assessment using KRS, or Kendall Reordering Score (Birch et al., 2010), which measures the similarity between the source-reference reorderings and the source-MT output reorderings.",
        "startOffset" : 115,
        "endOffset" : 135
      }, {
        "referenceID" : 11,
        "context" : "To compute the word alignments required by KRS, we used the FastAlign tool (Dyer et al., 2013).",
        "startOffset" : 75,
        "endOffset" : 94
      }, {
        "referenceID" : 2,
        "context" : "To put our results into perspective, note that Birch (2011) reports a difference of 5 KRS points between the translations of a PBMT system and those produced by four human translators tested against each other, in a Chinese-English experiment.",
        "startOffset" : 47,
        "endOffset" : 60
      } ],
      "year" : 2017,
      "abstractText" : "Within the field of Statistical Machine Translation (SMT), the neural approach (NMT) has recently emerged as the first technology able to challenge the long-standing dominance of phrase-based approaches (PBMT). In particular, at the IWSLT 2015 evaluation campaign, NMT outperformed well established state-ofthe-art PBMT systems on English-German, a language pair known to be particularly hard because of morphology and syntactic differences. To understand in what respects NMT provides better translation quality than PBMT, we perform a detailed analysis of neural versus phrase-based SMT outputs, leveraging high quality post-edits performed by professional translators on the IWSLT data. For the first time, our analysis provides useful insights on what linguistic phenomena are best modeled by neural models – such as the reordering of verbs – while pointing out other aspects that remain to be improved.",
      "creator" : "LaTeX with hyperref package"
    }
  }