###### Abstract  
Korean and Japanese have different writing scripts but share the same Subject-Object-Verb (SOV) word order. In this study, we pre-train a language-generation model using a Masked Sequence-to-Sequence pre-training (MASS) method on Korean and Japanese monolingual corpora. When building the pre-trained generation model, we allow the smallest number of shared vocabularies between the two languages. Then, we build an unsupervised Neural Machine Translation (NMT) system between Korean and Japanese based on the pre-trained generation model. Despite the different writing scripts and few shared vocabularies, the unsupervised NMT system performs well compared to other pairs of languages. Our interest is in the common characteristics of both languages that make the unsupervised NMT perform so well. In this study, we propose a new method to analyze cross-attentions between a source and target language to estimate the language differences from the perspective of machine translation. We calculate cross-attention measurements between Korean-Japanese and Korean-English pairs and compare their performances and characteristics. The Korean-Japanese pair has little difference in word order and a morphological system, and thus the unsupervised NMT between Korean and Japanese can be trained well even without parallel sentences and shared vocabularies. Keywords:MASS; pre-trained generation model; unsupervised neural machine translation; language typology; writing script; SOV word order +
Footnote †: journal: Medical  
are gathered into the Japanese and Korean monolingual corpora, respectively. By doing this, we allow the smallest number of common vocabularies in the NMT system between the two languages. Only a few punctuations are shared by the two corpora. We also build an unsupervised Korean-English NMT system using the same method. To compare with the Korean-Japanese NMT, the Korean-English NMT system is built to have as many common vocabularies between Korean and English as possible. Korean, Japanese, and English do not share writing scripts; however, Korean-Japanese and Korean-English have very different characteristics. Due to the geographic proximity, Japanese and Korean share considerable similarities in typological features of their syntax and morphology, while having different writing scripts [4; 5]. It is widely accepted that unsupervised NMT systems with high accuracies are possible only when the source and target language have a high lexical resemblance with the same scripts. Although the vocabularies shared by Korean and Japanese are only a few punctuations, the unsupervised NMT system can learn to translate between the two languages without a parallel corpus. Unlike a Korean-Japanese pair, the unsupervised Korean-English NMT system shows disappointing results because the language pair has a different morphology system and word order despite having as many common vocabularies as possible. These preliminary experiments show that the similarities of morphology and syntax between the two languages can be an essential prerequisite for successfully training an unsupervised NMT without common vocabularies. In this study, we want to discover what knowledge a model acquires through the pre-training process and what knowledge from the pre-trained model enables an unsupervised NMT with high accuracy. Therefore, we analyze which layers of the unsupervised NMT system store what kind of information, and whether cross-attentions demonstrate differences in the languages, such as word order. These findings can give us insight into how to train an unsupervised NMT system between various language pairs. The contributions of this paper are as follows:  
* We propose a new method to analyze cross-attentions of an encoder-decoder architecture considering the difference in properties between a source and target sequence. * We apply the analysis method to an unsupervised NMT task between Korean and Japanese, which are known to be very similar each other.