Valutazione umana di Google Traduttore e DeepL per le traduzioni di testi giornalistici dall’inglese verso l’italiano
Authors: Mirko Tavosanis (Università di Pisa, 2019)
Topic: Machine Translation

Abstract
The contribution presents an evaluation of the performance of Google Translator and DeepL through the web interfaces available to the public. For the evaluation was used a sample of 100 sentences taken from journalistic texts in English translated into Italian. The translations produced were evaluated by humans and the results of the evaluation were compared with the calculation of the BLEU score. The human evaluation of automatic systems showed quality levels close to those of human translation, while the BLEU score did not show close correspondence with human evaluation.

Introduction
Machine translation systems are playing an increasingly important role in everyday life, either alone or integrated into other practices (Bersani Berselli 2011). Their diffusion could also produce structural innovations and transform in depth some work sectors, beginning with the teaching of foreign languages (Ostler 2010; Tavosanis 2018). However, the assessment of the actual performance of these systems remains a complex problem both from a theoretical and from a practical point of view. Moreover, the difficulty of evaluation has long been considered one of the main constraints also for the development of translation systems (Pieraccini 2012, p. 275; Hajič 2008, p. 85). Numerous automatic or semi-automatic metrics have been developed for the evaluation; the most used in recent times has been BLEU (Papinieni e altri, 2002). The work on metrics is constant and, in particular, to the evaluation of metrics is dedicated one of the shared tasks of the WMT conferences (the results of the most recent are presented in Fourth Conference on Machine Translation 2019, pp. 494-525). However, over the last few years it has become apparent that the most used metrics are not actually able to adequately describe the differences and performance improvements of the latest machine translation systems, and in particular those based on neural networks. The problem can be described in general as a problem of poor correlation between metrics and human judgment. To use as a point of reference human judgment seems to be entirely correct from the methodological point of view: the objective of translation systems is mainly to provide translations that human beings consider to be of a good level. In particular, the non-perfect correlation between BLEU and human judgement has long been noted (e.g. Callison-Burch, Osborne and Koehn 2006) and several evaluators have reiterated the need to consider human evaluation as primary (Callison-Burch and others 2008, p. 72). The situation has probably been made less evident also by the frequent habit of evaluating different systems by comparing performance between them and not on an absolute scale; however, this practice has never been the only one and the systems presented in the main evaluation campaign on machine translation, the WMT tasks, are evaluated only with absolute judgements, not with relative judgements. The problem has been particularly evident in recent years, following the rapid introduction of neural network-based translation systems. BLEU, such as statistical translation systems (PB-SMT), bases its operation on n-grams. However, it is considered that this mechanism shows a bias against systems that adopt translation mechanisms not based on n-grams, such as neural network-based systems (Way 2018, p. 170). Several verifications have shown that in practice BLEU strongly underestimates the results of neural network translation systems (Bentivogli and others 2018a; Shterionov and others 2018). Of course, the validity of these tests can be related to the characteristics of specific samples. However, the data available today justify the idea that BLEU cannot be used as a general indicator of the quality of these systems. In this context there is no shortage of statements claiming the achievement of the "parity" between machine translation and human translation for some commercial systems (Hassan and others 2018). However, the independent verifications have not at the moment confirmed these results; on the contrary, they have shown significant differences (Toral and other 2018). Such statements however show the usefulness of an external evaluation of the performance of the most used systems. The present contribution also contributes to this activity, documenting the state of things for widely distributed products and in a context of real use for a language on which evaluations have so far been rather reduced. Some texts generated by automatic translation have therefore been subjected to human evaluation, together with products of human translators, with the aim of: 1. providing a human evaluation of the performance (absolute and non-relative) of two different systems 2. comparing the results of human evaluation with those of the evaluation obtained through BLEU.

Conclusion
The work described here is one of the first realizations of a broader project, dedicated to studying the possibilities of structural integration of automatic translators into the teaching practice of languages starting from the analysis of performance and the possibility of easily integrating products into the path of a translator in training. More extensive evaluations should therefore be available within a few months. For the Italian translation, these evaluations could be of particular interest, considering not only the speed of the recent improvements but also the fact that the Italian has been relatively little represented in the analyses carried out so far. For future developments, having considered a single textual genre, however varied, is a clear limit of analysis (Burchardt and others 2017: 159-160): the extension of the evaluation to different types compared to the newspaper or periodical article could easily lead to results very different from those described here. The inclusion of other textual genres is therefore undoubtedly the most important requirement in the design of an evaluation work on a larger scale. From this point of view, it seems particularly interesting to extend the work to specialist texts.