# Reusing Monolingual Pre-Trained Models by Cross-Connecting Seq2 seq Models for Machine Translation
## 5 Conclusions  
In this work, we proposed a novel method that efficiently reuses the pre-trained monolingual seq2 seq models for machine translation. Our method pre-trains several seq2 seq models with monolingual corpora independently, cross-connects them with an additional intermediate layer, and then fine-tunes the translation model on the parallel corpus. Because we can create \(n(n-1)\) translation pairs with \(n\) encoders and decoders, the monolingual pre-trained models can work as a multilingual pre-trained model. Their capacity is not affected by the number of languages because each model is pre-trained separately. We showed that the translation performance is improved significantly by our cross-connection method. We investigated the size and architecture of the intermediate layer and we found that the bigger parameters of the intermediate layer do not necessarily improve the performance. Furthermore, we showed that random initialization of decoder cross-attention is not optimal. When initializing the decoder, using pre-trained encoder weights is possible but using pre-trained decoder weights is better. The pre-trained decoder contributes to the performance improvement, although its effect is not as crucial as the encoder because the decoder depends on the representations that the encoder generates. Further, we found that the pre-trained decoder loses its effect when the fine-tuning data are plenty, and the cross-connection method is suitable for the low-resource machine translation. Our limitation is that the size of the model and the number of languages are limited due to the computational resources. For future work, we will extend our study to other language pairs and increase the number of languages.  
Conceptualization and methodology, J.O. and Y.-S.C.; investigation, J.O.; data curation and conceiving experiments, J.O.; performing experiments and designing results, J.O.; writing--original draft preparation, J.O.; writing--review and editing, J.O. and Y.-S.C.; supervision, Y.-S.C.; funding acquisition, Y.-S.C. All authors have read and agreed to the published version of the manuscript.  
This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No.2018R1A5A7059549), National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No.2020R1A2C1014037), and the Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No. 2020-0-01373, Artificial Intelligence Graduate School Program (Hanyang University)).  
Publicly available datasets were analyzed in this study. This data can be found here: [[[https://www.statmt.org/](https://www.statmt.org/)]([https://www.statmt.org/](https://www.statmt.org/)), accessed on 14 August 2021, [[https://sites.google.com/site/iwsltevaluation2014/data-provided](https://sites.google.com/site/iwsltevaluation2014/data-provided)]([https://sites.google.com/site/iwsltevaluation2014/data-provided](https://sites.google.com/site/iwsltevaluation2014/data-provided)), accessed on 14 August 2021, [[https://sites.google.com/site/iwsltevaluation2017](https://sites.google.com/site/iwsltevaluation2017)]([https://sites.google.com/site/iwsltevaluation2017](https://sites.google.com/site/iwsltevaluation2017)), accessed on 14 August 2021].  
The authors declare no conflict of interest.