# Improving the Performance of Vietnamese-Korean Neural Machine Translation with Contextual Embedding
## 1 Introduction  
Bidirectional encoder representations for Transformers, abbreviated as BERT, is a pre-trained model introduced by Google [1]. BERT has greatly improved the quality of natural language processing (NLP) tasks [1, 2, 3]. This state-of-the-art context-based embedding model comprises two tasks: masked-language modelling (MLM) and next-sentence prediction (NSP). MLM uses context words surrounding a masked word to predict what the masked word should be. When two sentences are fed into the BERT model, NSP predicts whether or not the second sentence can follow the first sentence.  
There are different variants of BERT for different languages: A Lite Bert (ALBERT) [4], Robustly Optimized BERT (RoBERTa) [5], and SpanBERT [6] are used for English; FlauBERT [7] for French; PhoBERT [3] for Vietnamese; GottBERT [8] for German; MacBERT for Chinese [9]; and KR-BERT [10] for Korean.  
The ALBERT variant uses cross-layer parameter-sharing and factorized embedding-layer parameterization techniques to decrease the number of parameters, thus reducing training time. RoBERTa, a pre-training approach, is one of the most popular variants of the BERT model. To improve the original BERT model, RoBERTa has implemented several changes in pre-training, such as using byte-level byte-pair encoding (BBPE) and dynamic masking, instead of the static masking used in the MLM task.  
PhoBERT is a pre-trained BERT variant for the Vietnamese language. It is based on the RoBERTa model. PhoBERT was trained on a huge Vietnamese corpus, including a 50 GB Vietnamese dataset.  
Neural machine translation (NMT) directly translates a source sentence to a target sentence by using a neural network [11; 12]. A series of publicly available toolkits, such as OpenNMT [13], Sockeye [14], and Fairseq [15], have made it easier for researchers to access NMT. NMT research is mostly performed on many rich-resource languages [2; 16; 17], but research in low-resource languages is receiving more attention [18; 19; 20].  
The neural network models used in studies are diverse. For example, [PERSON] et al. used Gap NMT (GNMT) [21]; [PERSON] et al. used convolutional sequence-to-sequence learning (ConvS2S) [22]; [PERSON] et al. used a hybrid translation model [23]; and [PERSON] et al. applied the transformer model [24].  
For Korean-Vietnamese machine translation (MT), [PERSON] et al. [25] published a dataset of more than 412,000 sentence pairs for training with the NMT model. [PERSON] et al. [26] performed a series of experiments with NMT by applying morphological analysis (MA) and word-sense disambiguation (WSD) to Korean sentences, before feeding language pairs into the MT system. In terms of TER [27] and BLEU [28] evaluation metrics, their system has led to the most significant improvement in the Korean-to-Vietnamese MT system by applying MA and WSD to Korean sentences and word segmentation to Vietnamese sentences. [PERSON] et al. [29] used the [PERSON] toolkit [30] to perform Korean-Vietnamese statistical MT (SMT). Before using a parallel corpus for training in the SMT model, they grouped component morphemes of adjectives and verbs in Korean sentences.  
In this research, we apply the transformer model for Korean-Vietnamese MT, in which the MT model combines the pre-trained contextual-embedding Vietnamese BERT model and NMT. Before being included in the MT systems, the Korean-Vietnamese bilingual corpus is prepared by a series of pre-treatment steps to improve the quality of the MT. In particular, we tagged POS [3] to Vietnamese sentences, and we applied MA and WSD to Korean sentences. This POS tagging was based on a Vietnamese pre-trained contextual-embedding model. We used BLEU, METEOR [31], and TER scores to evaluate the accuracy of MT systems.