Multilingual Neural Machine Translation for Low Resource Languages
Authors: Surafel M. Lakew, Mattia A. Di Gangi, Marcello Federico (Università di Trento, Fondazione Bruno Kessler, 2017)
Topic: Machine Translation

Abstract
Neural Machine Translation (NMT) has been shown to be more effective in translation tasks compared to the Phrase-Based Statistical Machine Translation (PBMT). However, NMT systems are limited in translating low-resource languages (LRL), due to the fact that neural methods require a large amount of parallel data to learn effective mappings between languages. In this work we show how so-called multilingual NMT can help to tackle the challenges associated with LRL translation. Multilingual NMT forces words and sub-words representation in a shared semantic space across multiple languages. This allows the model to utilize a positive parameter transfer between different languages, without changing the standard attention-based encoder-decoder architecture and training modality.} We run preliminary experiments with three languages (English, Italian, Romanian) covering six translation directions and show that for all available directions the multilingual approach, i.e. just one system covering all directions is comparable or even outperforms the single bilingual systems. Finally, our approach achieve competitive results also for language pairs not seen at training time using a pivoting (x-step) translation.

Introduction
Neural machine translation (NMT) has recently shown its effectiveness by delivering the best performance in various evaluation campaigns (IWSLT 2016 AUTHOR, WMT 2016 AUTHOR).} Unlike rule-based or phrase-based MT, the end-to-end learning approach of NMT models the mapping from source to target language directly through a posterior probability}. The basic component of an NMT system include an encoder, a decoder and an attention mechanism AUTHOR. Despite the continuous improvement in performance and translation quality, NMT models are highly dependent on the availability of large parallel} data, which in practice} can only} be acquired for a very limited number of language pairs}. For this reason,} building effective NMT systems} for low-resourced languages becomes a} primary challenge AUTHOR. Recently, AUTHOR showed how a standard string-to-tree statistical MT system AUTHOR can effectively outperform NMT methods for low-resource} languages, such as Hausa, Uzbek, and Urdu. In this work, we focus on a so-called multilingual NMT AUTHOR, which considers the use of NMT to target many-to-many language translation. Our motivation is that intensive cross-lingual transfer AUTHOR via parameter sharing should ideally help in the case of similar languages and sparse training data. Hence, in this work we investigate multilingual NMT across Italian, Romanian, and English, and simulate low-resource conditions by limiting the amount of parallel data. Our approach showed a BLEU increase in various language directions, in a low-resource setting. To compare a single language pair NMT models with a single multilingual NMT (M-NMT) model, we considered six translation directions (i.e English\leftrightarrowItalian, English\leftrightarrowRomanian, and Italian\leftrightarrowRomanian). For evaluating the zero-shot translation (i.e. a translation between language pair with no available parallel corpus), we removed the (Italian\leftrightarrowRomanian) language pairs. In the same way as the six-language-pairs, the performance of the four-language-pairs M-NMT model is comparable with the bilingual models for the language directions with parallel data. We start in Section 2 with a brief description of NMT and state-of-the-art multilingual NMT approaches. In Section 3, we give a background on our M-NMT model. In Section 4, we present the experimental setting and the NMT model configurations. In Section 5, we show and discuss the results of the experiments. Finally, in Section 6 we present our conclusion and future works.

Conclusion
In this paper, we used a multilingual NMT model in a low-resource language pairs scenario. We showed that a single multilingual system achieves comparable performances with the bilingual baselines while avoiding the need to train several single language pair models. Then, we showed how a multilingual model can be used for zero-shot translation by using a pivot language for achieving slightly lower results than a bilingual model trained on that language pair. As a future work we want to explore how the choice of different languages can enable a better parameter transfer in a single model, using more linguistic features of the surface word form, and how to achieve a direct zero-shot translation in a low-resource scenario without the pivoting mechanism.