## 3 Pre-Trained Models and Unsupervised NMTs
### Datasets and Pre-Training Setup
#### 3.1.1 Monolingual Corpora  
**(K)**: The Korean corpus is collected from the Korean Contemporary Corpus of Written Sentences ([[http://nlp.kookmin.ac.kr/kcc/](http://nlp.kookmin.ac.kr/kcc/)]([http://nlp.kookmin.ac.kr/kcc/](http://nlp.kookmin.ac.kr/kcc/)), accessed on 19 August 2021). To not only maximize lexical overlap with the English corpus but also minimize lexical overlap with Japanese corpus, we collect sentences including as many alphabets and digitsas possible, but not Chinese characters. They are tokenized using the ETRI parser ([[https://aiopen.etri.re.kr/service_api.php](https://aiopen.etri.re.kr/service_api.php)]([https://aiopen.etri.re.kr/service_api.php](https://aiopen.etri.re.kr/service_api.php)), accessed on 19 August 2021).
* The Japanese corpus is collected from WMT2019 ([[https://www.statmt.org/wmt1/translation-task.html](https://www.statmt.org/wmt1/translation-task.html)]([https://www.statmt.org/wmt1/translation-task.html](https://www.statmt.org/wmt1/translation-task.html)), accessed on 19 August 2021) and JParaCrawl (2014). To minimize lexical overlap with the Korean corpus, we only collect sentences with neither alphabets nor digits. They are tokenized using the Mecab tokenizer ([[https://taku910.github.io/mecab/](https://taku910.github.io/mecab/)]([https://taku910.github.io/mecab/](https://taku910.github.io/mecab/)), accessed on 19 August 2021).
* The English corpus is collected from WMT 2017-2019 datasets ([[https://www.statmt.org/wmt17/translation-task.html](https://www.statmt.org/wmt17/translation-task.html)]([https://www.statmt.org/wmt17/translation-task.html](https://www.statmt.org/wmt17/translation-task.html));[[https://www.statmt.org/wmt18/translation-task.html](https://www.statmt.org/wmt18/translation-task.html)]([https://www.statmt.org/wmt18/translation-task.html](https://www.statmt.org/wmt18/translation-task.html));[[https://www.statmt.org/wmt19/translation-task.html](https://www.statmt.org/wmt19/translation-task.html)]([https://www.statmt.org/wmt19/translation-task.html](https://www.statmt.org/wmt19/translation-task.html)), accessed on 19 August 2021), and those sentences are tokenized by the Moses tokenizer ([[https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl](https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl)]([https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl](https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl))), accessed on 19 August 2021) for pre-processing.  
These corpora consist of 5 million sentences each. They are all used in building the pre-trained generation models.  
First, we build a monolingual lexicon for each corpus separately using the BPE algorithm ([PERSON] et al., 2019) fixed at 60,000 sub-word units. The results are shown in the first column of Table 1. Then, we build a joint lexicon between the two languages with the same fixed number of sub-word units using the BPE algorithm. The numbers of (shared) vocabularies in the joint lexicons are shown in the second column of Table 1. Because the Japanese corpus does not contain alphabets and digits, the number of shared vocabularies of Korean-Japanese is only 33, most of which are punctuations. On the contrary, since most Korean sentences contain alphabets and digits, the ratio of the shared vocabulary of Korean-English rises to more than 30% despite the use of different writing scripts.