Online and Multitask learning for Machine Translation Quality Estimation in Real-world scenarios
Authors: Jose' Guilherme Camargo De Souza, Marco Turchi, Matteo Negri, Antonios Anastasopoulos (Fondazione Bruno Kessler, Università di Trento, University of Notre Dame, 2014)
Topic: Machine Translation

Abstract
We investigate the application of different supervised learning approaches to machine translation quality estimation in realistic conditions where training data are not available or are heterogeneous with respect to the test data. Our experiments are carried out with two techniques: on-line and multitask learning. The former is capable to learn and self-adapt to user feedback, and is suitable for situations in which training data is not available. The latter is capable to learn from data com-ing from multiple domains, which might considerably differ from the actual testing domain. Two focused experiments in such challenging conditions indicate the good potential of the two approaches.

Introduction
Quality Estimation (QE) for Machine Translation (MT) is the task of estimating the quality of a translated sentence at run-time and without access to reference translations (Specia et al., 2009; Soricut and Echihabi, 2010; Bach et al., 2011; Specia, 2011; Mehdad et al., 2012; C. de Souza et al., 2013; C. de Souza et al., 2014a). As a quality indicator, in a typical QE setting, automatic systems have to predict either the time or the number of editing operations (e.g. in terms of HTER1) required to a human to transform the translation into a syntactically/semantically correct sentence. In recent years, QE gained increasing interest in the MT community as a possible way to: i) decide whether a given translation is good enough for publishing as is, ii) inform readers of the target language only whether or not they can rely on a translation, iii) filter out sentences that are not good enough for post-editing by professional translators, or iv) select the best translation among options from multiple MT and/or translation memory systems. So far, despite its many possible applications, QE research has been mainly conducted in controlled lab testing scenarios that disregard some of the possible challenges posed by real working conditions. Indeed, the large body of research resulting from three editions of the shared QE task organized within the yearlyWorkshop on Machine Translation (WMT – (Callison-Burch et al., 2012; Bojar et al., 2013; Bojar et al., 2014)) has relied on simplistic assumptions that do not always hold in real life. These assumptions include the idea that the data available to train QE models is: i) large (WMT systems are usually trained over datasets of 800/1000 instances) and ii) representative (WMT training and test sets are always drawn from the same domain and are uniformly distributed).In order to investigate the difficulties of training a QE model in realistic scenarios where such conditions might not hold, in this paper we approach the task in situations where: i) training data is not available at all (x2), and ii) training instances come from different domains (x3). In these two situations, particularly challenging from the machine learning perspective, we investigate the potential of online and multitask learning methods (the former for dealing with the lack of data, and the latter to cope with data heterogeneity), comparing them with the batch methods currently used.

Conclusion
We investigated the problem of training reliable QE models in particularly challenging conditions from the learning perspective. Two focused experiments have been carried out by applying: i) online learning to cope with the lack of training data, and ii) multitask learning to cope with heterogeneous training data. The positive results of our experiments suggest that the two paradigms should be further explored (and possibly combined) to overcome the limitations of current methods and make QE applicable in real-world scenarios.