Can Monolingual Embeddings Improve Neural Machine Translation?
Authors: Mattia A. Di Gangi, Federico Marcello (Università di Trento, Fondazione Bruno Kessler, 2017)
Topic: Machine Translation

Abstract
Neural machine translation (NMT) %notoriously: used to emphasize that a quality or fact, <<typically a bad one>>, is well known: the company is notoriously difficult to contact | a notoriously overcrowded prison. recently redefined the state of the art in machine translation, by introducing deep learning architecture that can be trained end-to-end. One limitation of NMT is the difficulty to learn representations of rare words. The most common solution is to segment words into sub-words, in order to allow for shared representations of infrequent words. In this paper we present ways to directly feed a NMT network with external word embeddings trained on monolingual source data, thus enabling a virtually infinite source vocabulary. Our preliminary results show that while our methods do not seem effective under large-data training conditions (WMT En-De), they instead show great potential for the typical low-resourced data scenario (IWSLT En-Fr). By leveraging external embeddings learned on Web crawled English texts, we were able to improve a word-level En-Fr baseline trained on 200,000 sentence pairs by up to 4 BLEU points.

Introduction
The latest developments of machine translation have been led by the neural approach (AUTHOR; AUTHOR), a deep-learning based technique that has shown to outperform the previous methods in all the recent evaluation campaigns (AUTHOR; AUTHOR). NMT mainly relies on parallel data, which are expensive to produce as they involve human translation. Recently, back-translation AUTHOR has been proposed to leverage target language data. This consists in enriching the training data with synthetic translations produced with a reverse MT system AUTHOR. Unfortunately, this method introduces noise and seems really effective only when the synthetic parallel sentences are only a fraction of the true ones. Hence, this approach does not allow to leverage huge quantities of monolingual data. One consequence of the scarcity of parallel data is the occurrence of out-of-vocabulary (OOV) and rare words. In fact, being NMT a statistical approach, it cannot learn meaningful representations or rare words and no representation at all for OOV words. The solution up to this moment is to segment words into sub-words (AUTHOR; AUTHOR) in order to have a better representation of rare and OOV words, as parts of their representation will be ideally shared with other words. The drawback of this approach is that it generates longer input sequences, thus exacerbates the handling of long-term dependencies (AUTHOR). In this paper, we propose to keep the source input at a word level while alleviating the problem of rare and OOV words. We do it by integrating the usual word indexes with word embeddings that have been pre-trained on huge monolingual data. The intuition is that the network should learn to use the provided representations, which should be possibly more reliable for the rare words. This should be true particularly for the low-resource settings, where pa- rameter transfer has shown to be an effective approach (AUTHOR). Because of the softmax layer, the same idea cannot be applied straightforwardly to the target side, hence we continue to use sub-words there. We show that the network is capable to learn how to translate from the input embeddings while replacing the source embedding layer with a much smaller feed-forward layer. Our results show that this method seems effective in a small training data setting, while it does not seem to help under large training data conditions. In the following section we briefly describe the state-of- the-art NMT architecture. Then, we introduce our modification to enable the use of external word embeddings. In Section 4, we introduce the experimental setup and show our results, while in Section 5 we discuss our solution. Finally, in Section 6 we presents our conclusions and the future work.

Conclusion
In this paper we propose three methods to extend the input word embeddings to an NMT network in order to leverage a word representation coming from a big monolingual corpus. Our results show that this approach greatly improves over an NMT baseline in a low-resource scenario, while it is not helpful for better-resourced tasks. Using monolingual data for improving NMT is a problem also in the latter case, thus our future work will focus on how to integrate models larger than word embeddings, and trained on monolingual data, to improve word and sentence representations.