# Reusing Monolingual Pre-Trained Models by Cross-Connecting Seq2 seq Models for Machine Translation
## 1 Introduction  
Transfer learning with pre-training and fine-tuning has pushed the state-of-the-art results in many natural language processing (NLP) tasks since the famous success of BERT [1]. This method now has become a common practice in the NLP field. The unsupervised pre-training on unlabeled data is particularly beneficial when the labeled training data are limited. The neural machine translation (NMT) task is the case because parallel corpora are expensive to construct and not easily available for many languages, while monolingual corpora are relatively easier to find and large in scale. Therefore, we expect a translation model's performance to be improved by pre-training using rich monolingual data and fine-tuning with parallel data. However, when using transfer learning for NMT, there are two issues. First, NMT is a cross-lingual task, which involves at least two languages. Recent approach for the cross-lingual tasks is to pre-train a model with multilingual objective [2; 3; 4], but there are also several monolingual pre-trained models of various languages, such as [5] for Brazilian [6; 7], for Dutch [8], for Romanian [9], for Russian [10; 11; 12], for French [13; 14], for Italian, and [15] for Portuguese. In multilingual models, each language has only a limited allocation for model capacity, unlike in monolingual models. Furthermore, the monolingual models are relatively easier to create. Therefore, it would be beneficial if the monolingual models can be reused for the machine translation task. Another issue is that NMT is a sequence generation task. Natural language generation (NLG) tasks, including NMT, usually need both the encoder and the decoder. It can be thought that both the pre-trained encoder and decoder would improve the performance of a language generation model. However, reuse of a whole pre-trained sequence-to-sequence (seq2 seq) [16] model is not very common. Existing approaches usually leverage only a part of a seq2 seq model, namely an encoder such as BERT and XLM [17], and a decoder such as GPT [18]. Our purpose is to reuse the monolingual seq2 seq models pre-trained for monolingual tasks, such as classification or summarization, also for the machine translation task. Yet, a monolingual pre-trained seq2 seq model is not fully appropriate for NMT since it is a cross-lingual task. For this, we propose the 'cross-connection', a simple method that use monolingual pre-trained seq2 seq models for NMT. In this paper, we pre-train whole seq2 seq models with monolingual corpora, then we construct a translation model by 'cross-connecting' a seq2 seq of one language with another seq2 seq of a different language, and then we fine-tune the translation model with a parallel corpus. For pre-training, each model receives inputs where some tokens are dropped, and it is trained for each language to predict the dropped tokens. For fine-tuning, a seq2 seq model is initialized by the encoder of the source language model and the decoder of the target language model, and it is fine-tuned on parallel data for translation. A remaining problem is that the encoder and the decoder are pre-trained completely separately for different languages, and do not know which language they would be composed with. For mediation between these two modules, we insert an additional 'intermediate layer'. This layer helps mapping of the independently pre-trained encoder and decoder. Because they are composable, our monolingual pre-trained models can work as a multilingual pre-trained model. A seq2 seq model pre-trained on one language can be cross-connected with another model that is pre-trained on any other language. For example, if we have three pre-trained seq2 seq models of English, Romanian, and German, we can reuse them to the English-German, English-Romanian, German-Romanian translation models. If we train another seq2 seq model on Italian, we can reuse them to the English-Italian, German-Italian, Romanian-Italian translation models. It is possible to create \(n(n-1)\) translation pairs with \(n\) encoders and decoders.