On the Development of Customized Neural Machine Translation Models
Authors: Mauro Cettolo, Roldano Cattoni, Marco Turchi (Fondazione Bruno Kessler, 2021)
Topic: Machine Translation

Abstract
Recent advances in neural modeling boosted performance of many machine learning applications. Training neural networks requires large amounts of clean data, which are rarely available; many methods have been designed and investigated by researchers to tackle this issue. As a partner of a project, we were asked to build translation engines for the weather forecast domain, relying on few, noisy data. Step by step, we developed neural translation models, which outperform by far Google Translate. This paper details our approach, that - we think - is paradigmatic for a broader category of applications of machine learning, and as such could be of widespread utility.

Introduction
The field of machine translation (MT) has experienced significant advances in recent years thanks to improvements in neural modeling. On the one hand, this represents a great opportunity for industrial MT, on the other it also poses the great challenge of collecting large amounts of clean data, needed to train neural networks. MT training data are parallel corpora, that is collections of sentence pairs where a sentence in the source language is paired with the corresponding translation in the target language. Parallel corpora are typically gathered from any available source, in most cases the web, without much guarantees about quality nor domain homogeneity. Over the years, the scientific community has accumulated a lot of knowledge on ways to address the problem of the quantitative and qualitative inadequacy of parallel data necessary to develop translation models. Among others, deeply investigated methods are: corpus filtering <cit.>, data augmentation such as data selection <cit.> and back-translation <cit.>, model adaptation <cit.>. They should be the working tools of anyone who has to develop neural MT models for specific language pairs and domains. This paper reports on the development of neural MT models for translating forecast bulletins from German into English and Italian, and from Italian into English and German. We were provided with in-domain parallel corpora for each language pair but not in sufficient quantity to train a neural model from scratch. Moreover, from the preliminary analysis of data, the English side resulted noisy (e.g. missing or partial translations, misaligned sentences, etc.), affecting the quality of any pair involving that language. For this very reason, we focus on one of the pairs involving English we had to cover, namely Italian-English. An overview of the in-domain data and the description of their analysis are given in Section <ref>, highlighting the issues that emerged. Section <ref> describes the previously listed methods together with their employment in our specific use-case. Developed neural translation models are itemized in Section <ref>, where their performance are compared and discussed; our best models outperform by far Google Translate and some examples will give a grasp of the actual translation quality. We think that our approach to the specific problem we had to face is paradigmatic for a broader category of machine learning applications, and we hope that it will be useful to the whole NLP scientific community.

Conclusion
In this paper we described the development process that led us to build competitive customized translation models. Given the provided in-domain data, we started by analyzing them under several perspectives and discovered that they are few, noisy and heterogeneous. We faced these issues by exploiting a number of methods which represent established knowledge of the scientific community: adaptation of neural models, corpus filtering and data augmentation techniques such as data selection and back-translation. In particular, corpus filtering allowed us to avoid the misleading results observed on the original noisy data, while adaptation and data augmentation proved useful in effectively taking advantage of out-of-domain resources.