Abstract: This paper describes CAiRE's submission to the unsupervised machine translation track of the WMT'19 news shared task from German to Czech. We leverage a phrase-based statistical machine translation (PBSMT) model and a pre-trained language model to combine word-level neural machine translation (NMT) and subword-level NMT models without using any parallel data. We propose to solve the morphological richness problem of languages by training byte-pair encoding (BPE) embeddings for German and Czech separately, and they are aligned using MUSE (Conneau et al., 2018). To ensure the fluency and consistency of translations, a rescoring mechanism is proposed that reuses the pre-trained language model to select the translation candidates generated through beam search. Moreover, a series of pre-processing and post-processing approaches are applied to improve the quality of final translations.

Introduction: Machine translation (MT) has achieved huge advances in the past few years BIBREF1, BIBREF2, BIBREF3, BIBREF4. However, the need for a large amount of manual parallel data obstructs its performance under low-resource conditions. Building an effective model on low resource data or even in an unsupervised way is always an interesting and challenging research topic BIBREF5, BIBREF6, BIBREF7. Recently, unsupervised MT BIBREF8, BIBREF9, BIBREF0, BIBREF10, BIBREF11, which can immensely reduce the reliance on parallel corpora, has been gaining more and more interest. Training cross-lingual word embeddings BIBREF0, BIBREF12 is always the first step of the unsupervised MT models which produce a word-level shared embedding space for both the source and target, but the lexical coverage can be an intractable problem. To tackle this issue, BIBREF13 provided a subword-level solution to overcome the out-of-vocabulary (OOV) problem. In this work, the systems we implement for the German-Czech language pair are built based on the previously proposed unsupervised MT systems, with some adaptations made to accommodate the morphologically rich characteristics of German and Czech BIBREF14. Both word-level and subword-level neural machine translation (NMT) models are applied in this task and further tuned by pseudo-parallel data generated from a phrase-based statistical machine translation (PBSMT) model, which is trained following the steps proposed in BIBREF10 without using any parallel data. We propose to train BPE embeddings for German and Czech separately and align those trained embeddings into a shared space with MUSE BIBREF0 to reduce the combinatorial explosion of word forms for both languages. To ensure the fluency and consistency of translations, an additional Czech language model is trained to select the translation candidates generated through beam search by rescoring them. Besides the above, a series of post-processing steps are applied to improve the quality of final translations. Our contribution is two-fold: We propose a method to combine word and subword (BPE) pre-trained input representations aligned using MUSE BIBREF0 as an NMT training initialization on a morphologically-rich language pair such as German and Czech. We study the effectiveness of language model rescoring to choose the best sentences and unknown word replacement (UWR) procedure to reduce the drawback of OOV words. This paper is organized as follows: in Section SECREF2, we describe our approach to the unsupervised translation from German to Czech. Section SECREF3 reports the training details and the results for each steps of our approach. More related work is provided in Section SECREF4. Finally, we conclude our work in Section SECREF5.

Question: How is the quality of the translation evaluated?
Answer: They report the scores of several evaluation methods for every step of their approach.