Valutazione umana di DeepL a livello di frase per le traduzioni di testi specialistici dall'inglese verso l'italiano
Authors: Sirio Papa, Mirko Tavosanis (Università di Pisa, 2020)
Topic: Machine Translation

Abstract
The contribution presents an evaluation of the performance of DeepL in the translation of specialist texts from English to Italian. The evaluation was carried out at sentence level, on a sample of 108 phrases from environmental, energy, biomedicine and drug disciplines texts, and the translations produced were evaluated by trained translators with disciplinary skills. DeepL's translation obtained a statistically equal assessment of human translation in terms of adequacy and slightly lower in terms of smoothness. The automatic translation of texts has also received a higher score than that obtained, in similar ways, by the automatic translation of journalistic texts.

Introduction
The evaluation of the actual performance of machine translation systems continues to be a complex problem both from a theoretical and from a practical point of view. From a practical point of view, it is now evident that the most widely used measurement metrics after the 2000's, and in particular BLEU, are not actually able to adequately describe the differences and performance improvements of the systems currently in use, and in particular those based on neural networks (Bentivogli and others 2018a; Shterionov and others 2018; Tavosanis 2019). More recent proposals, such as BERTScore, have yet to be thoroughly evaluated and seem to provide very similar results to BLEU (Zhang and other 2020). It was therefore considered methodologically appropriate not to use them for this assessment, even as a comparison term. The only satisfactory type of assessment, to date, remains that carried out by human evaluators. Not all types of human assessment are equally satisfactory and reliable. The assessments carried out through crowdsourcing by individuals whose skills are not known assign for example to machine translation, systematically, higher scores than those awarded by persons with proven competence in the evaluation of translations (Castilho and others 2017a; Läubli and others 2020: 658). In this context there is no shortage of statements claiming the achievement of the "parity" between machine translation and human translation for some commercial systems (Hassan and others 2018). The independent verifications in some cases confirmed these results, but in others they showed significant differences (Toral and other 2018). This uncertainty is also largely due to the circumstances of the assessment, which can be very different from each other. This contribution aims to provide further results by entering into the research path described in Tavosanis (2019), against which it reinforces the method of investigation and varies the textual typology of reference. In Tavosanis (2019) the evaluations were carried out on journalistic texts; in this contribution specialized texts were chosen. The evaluation aims primarily to assess the quality of specialist translations in themselves and secondly to see whether the scores assigned to specialist translations are higher or lower than those assigned to translations of journalistic texts. In fact, automatic translation is normally used on texts belonging to very different genres, and evaluating a single genre is certainly very limiting. (Burchardt and others 2017: 159-160). In particular, given the significant linguistic differences between specialist texts and non-specialist texts, it seems likely that the same translation technology can produce very different results in both cases. Ensuring the quality of translations of texts from different domains has therefore been considered a problem since the first diffusion of systems based on neural networks. Koehn and Knowles (2017: 29), for example, noting that "in different domains, words have different translations and meaning is expressed in different styles" present the domain mismatch as the first "challenge" for these systems: in their assessments, in these contexts the NMT achieved results lower than those of the SMT systems. It also seems widespread that a generalist neural network translation system produces low quality results when applied to specialist texts (Chu and Wang 2020: 413). The domain adaptation is therefore an important development sector for the NMT (Chu and Wang 2018). However, assessments carried out with human evaluators are rare and almost absent for Italian.

Conclusion
The results obtained with this test support the hypothesis that even for Italian, at least for some textual types and at sentence level, automatic translation has reached a quality level statistically equal to that of human translation with regard to adequacy and slightly lower with regard to smoothness. They are therefore consistent with several other recent results, presented for other languages (Läubli and other 2020: 660), but it should be remembered that Italian was not included in the important tasks of WMT 2019 (Barrault and others 2019). In addition, the results obtained do not support the assumption that the NMT of general use systems achieves lower results when applied to specialist texts than when applied to non-specialized texts. The analysis naturally has several limits: for example, the assessed sample is relatively small, the fluctuations in the evaluators' judgement cannot be compared with an experienced professional average and the specialist domains taken into account are only two. However, the extension and improvement of these practices seem to be the only way to properly assess the capabilities of machine translation in Italian. As far as future developments are concerned, the need for a realistic assessment seems to make the transition from the evaluation of individual sentences to that of whole texts indispensable. The quality of machine translation at text level is, in many cases, significantly worse than that at sentence level (Läubli and others 2020: 660). The lack of structural systems to ensure text consistency in translation to neural networks suggests that the phenomenon is structural; however, dedicated assessments are needed to verify these hypotheses. At the same time, the comparison with the evaluation of journalistic texts suggests the idea that the results can vary significantly from one textual genre to another, and that at least in some cases they can be better than those obtained with non-specialist texts. Gender-related variability is not included in the very detailed summary of Läubli and others (2020), but it seems essential to take it structurally into account in order to make all future evaluations more robust.