Towards Deeper MT - A Hybrid System for German

The idea to improve MT quality by using deep linguistic and knowledge-driven information has frequently been expressed. If the goal is to use deep information for building an MT system, there are two extreme options: (1) to start from a purely knowledge-driven approach (RBMT) and try to arrive at the same recall found in current SMT systems; (2) to start from an SMT system and try to arrive at higher precision by modifying it so that more knowledge drives the translation process. The system architecture we will describe in this paper starts in the middle of these extreme options. It is a hybrid architecture that we take as a starting point for future experiments and extensions to increase MT quality by more knowledge-driven processing. 1 Introduction Statistical Machine Translation (SMT) based on comparably shallow features can be considered the most successful paradigm in Machine Translation (MT). The processing pipelines and machine learning architectures have become quite sophisticated and complex and allow for many types of optimisations. SMT systems have a (theoretical) high recall in the sense that they provide output for most input and that the pieces that would constitute a good translation are usually present somewhere in a huge search space during the translation process (e.g. in phrase tables or language models). However, it is very difficult to arrive at high precision, i.e., to automatically choose the right pieces and put together a fluent and accurate translation of a given input. The idea to further improve MT quality by adding deeper (i.e., more linguistic and knowledge-driven) information has thus frequently been expressed. At the same time, rule-based MT systems that primarily apply such knowledge and that are able to control precision much better are used only in certain nieches today. The reason is that they lack recall: for example, parsing failure or gaps in the lexicon typically lead to a dead-end where the only option is to manually code the missing information, which is too resource intensive especially if one wants to take care of those less frequent items and phenomena in the “long tail”. If one has the goal to use deep information for building an MT system with the best possible results, there are two extreme options: (1) to start from a purely knowledge-driven approach and try to arrive at the same (theoretical) recall found in current SMT systems; (2) to start from an SMT system and try to arrive at high precision by modifying it so that knowledge drives the search process. Today, it is an open research question what will lead to the best results in the end. The system architecture we will describe below starts in the middle of both extreme options. It is a hybrid architecture that we take as starting point for future experiments and extensions to increase MT quality by more knowledge-driven processing. This systems has been developed within the QTLeap project1 where it serves as a “deeper” baseline system as compared to a pure SMT baseline. The goal of the project is to explore different combinations of shallow and deep processing for improving MT quality. The system presented in this paper is the first of a series of system prototypes developed in the project. We therefore refer to it as System 1 in this paper. This work is licensed under a Creative Commons Attribution 4.0 International Licence. Page numbers and proceedings footer are added by the organisers. Licence details: http://creativecommons.org/licenses/by/4.0/ http://qtleap.eu/ Figure 1: Architecture of System 1. 2 A Hybrid System Combination for German↔English The fact that German is relatively well-resourced in comparison to other language pairs has allowed MT researchers to build strong statistical systems with very good performance on a lexical or a local level (Bojar et al., 2014). The German-English MT system we present here (System 1) aims to effectively incorporate deep linguistic processing into existing successful machine translation methods for this language pair. Since our main goal is to achieve a high-quality system that allows for experimentation, competes with state-of-the-art systems, and can be useful in the projects real use-case scenario (translating user queries and expert answers in a chat-based PC helpdesk scenario), we use a system implementation that takes advantage of deep transfer and also includes a statistical mechanism that enhances performance by keeping the best parts from each employed method. Figure 1 shows the overall hybrid architecture that includes: • A statistical Moses system, • the commercial transfer-based system Lucy, • their serial system combination, and • an informed selection mechanism (“ranker”). The components of this hybrid system will be detailed in the sections below. 2.1 Statistical MT system: Moses Our statistical machine translation component was based on a vanilla phrase-based system built with Moses (Koehn et al., 2007) trained on the following corpora: Europarl ver. 7, News Commentary ver. 9 (Bojar et al., 2014), Commoncrawl (Smith et al., 2013), and MultiUN (Eisele and Chen, 2010) as well as on the following domain corpora: the Document Foundation (Libreoffice Help – 47K sentence pairs, Libreoffice User Interface – 35K parallel entries), the Document Foundation Terminology (690 translated terms), the Document Foundation Website (226 sentence pairs), Chromium browser (6,3K parallel entries), Ubuntu Documentation (6,3K sentence pairs), Ubuntu Saucy (183K parallel entries), and Drupal web-content management (5K parallel entries). Language models of order 5 have been built and interpolated with SRILM (Stolcke, 2002) and KenLM (Heafield, 2011). For German to English, we also experimented with the method of pre-ordering the source side based on the target-side grammar (Popovic and Ney, 2006). As a tuning set we used the news-test 2013. In our architecture, this system on its own also serves as baseline. 2.2 Transfer-based MT system: Lucy The transfer-based core of System 1 is based on the Lucy system (Alonso and Thurmair, 2003) that includes the results of long linguistic efforts over the last decades and that has successfully been used in previous projects including Euromatrix+ and QTLaunchPad. The transfer-based approach has shown good results that compete with pure statistical systems, although its focus is on translating according to linguistic structures sets. Translation occurs in three phases, namely analysis, transfer, and generation. All three phases consist of hand-coded linguistic rules which have shown to perform well for capturing the structural and semantic differences between German and other languages. During the analysis phase, a parsing algorithm constructs a tree of the source language using a monolingual lexicon and the included grammar rules. The analysis algorithm reverts to a shallower analysis at the phrasal level in cases when the engine is not able to process the full tree. The analysis tree is subsequently used for the transfer phase, where deep representations of the source are transferred into deep representations of the target language using a bilingual lexicon based on canonical forms and categories. The generation phase creates the target sentence on the lexical level, using inflection and agreement rules between the dependent target language structures. A RestAPI allows the different processing steps and/or intermediate results to be influenced. Deep features for empirical enhancement Although deep techniques indicate good coverage of a number of linguistic phenomena, each of the three phases may frequently encounter serious robustness issues and/or the inability to fully process a given sentence. Erroneous analysis from early phases may aggregate along the pipeline and cause further sub-optimal choices in later phases, thus severely deteriorating the quality of the produced translation. Preliminary analysis (Federmann and Hunsicker, 2011) has shown that such is the case for source sentences that are ungrammatical in the first place or that have a very shallow syntax with many specialized lexical entries. To tackle these issues, we combine the transfer-based component with our supportive SMT engine in the following two ways: (a) train a statistical machine translation to automatically post-edit the output of the transfer-based system (“serial combination”) (b) use the post-edited or the SMT output in cases where the transfer-based system exhibits lower performance. This is done through an empirical selection mechanism that performs real-time analysis of the produced translations and automatically selects the output that is predicted to be of a better quality (Avramidis, 2011). Figure 1 shows the overall architecture of System 1 for en→de. 2.3 Serial System Combination: Lucy+Moses For automatic post-editing of the transfer-based system, a serial Transfer+SMT system combination is used, as described in (Simard et al., 2007) The first stage is translation of the source-language part of the training corpus by the transfer-based system. The second stage is training an SMT system with the transfer-based translation output as a source language and the target-language part as a target language. Later, the test set is first translated by the transfer-based system, and the obtained translation is translated by the SMT system. Figure 2 illustrates the architecture for translation direction en→de. Note that the notion of “German*” in the figure is meant to distinguish the input and output of the SMT system. “German*” is the normal output of the transfer-based system. In this linear system combination, improvement of up to 6 absolute BLEU points has been achieved for both translation directions in several pilot evaluations. Nevertheless, the method on its own could not outperform the SMT system trained on a large parallel corpus. The example in Figure 1 nicely illustrates how the statistical post-editing operates. While the original SMT output us