# Improving the Performance of Vietnamese-Korean Neural Machine Translation with Contextual Embedding
## 4 Experimentations and Results
### Discussion  
In the KrUTagger system, the MA task reduces the number of rare words by parsing words into sub-words. For example, the sentence \"_tai chi biong da gioi_\" (I play football **well**) in Vietnamese can be translated into \"_na-neun chug-gu-leul **jal-han-da**_\" or \"_na-neun chug-gu-leul **jal-hab-ai-da**_\" or \"_na-neun chug-gu-leul **jal-hae-yo**_\". If the candidate sentence is \"_na-neun chug-gu-leul **jal-han-da**_\" and the reference sentence is \"_na-neun chug-gu-leul **jal-hae-yo**_\", the meanings of the candidate and the reference are similar, but the evaluation method understands that \"_jal-hae-yo_\" and \"_jal-han-da_\" are different. Therefore, the result of the MT system is lower. With MA, the \"_jal-hae-yo_\" is split into \"_jal_\", \"_hae_\" and \"_yo_\"; \"_jal-han-da_\" is split into \"_jal_\", \"_han_\", and \"_da_\". When comparing two of these words, the first token of the two words is the same, thus leading to a better result under the MT system. In addition, the WSD task tags words with differing sense codes, depending on their meaning in different contexts. For instance, the word \"_sseu-da_\" in Korean has several meanings, such as \"_write_\", \"_wear_\", or \"_bitter_\". When translating a sentence containing the word \"_sseu-da_\", the MT system confuses the meaning of the word \"_sseu-da_\" in certain contexts. Our WSD tool converts \"_sseu-da_\" into \"_sseu-da_01_\", \"_sseu-da_02_\", or \"_sseu-da_03_\", depending on the context. By adding more information to Korean words, the Vietnamese-Korean MT system provides better results. In addition, in the VnPOS system, there are numerous polysemous words in Vietnamese. Many words have different meanings or diverse grammatical categories, depending on their context. For example, the word \"_lloc_\" can be a verb (\"to learn\") or it can be a noun (\"learning\"), depending on the sentence. To overcome this ambiguity, annotation POS is used to add information to Vietnamese sentences. The word \"_lloc_\" is transferred into \"_lloc_1_\" or \"_lloc_1_\". The POS tagging indicates the syntactic function of words, thus enabling the MT systemto find words in the target language more easily, with the same meanings and functions. As a result, POS enables the machine to translate more accurately. In the VietBERT+NMT system, the Vietnamese BERT model was used to extract representation for Vietnamese sentences. Unlike context-free models, such as word2 vec, BERT can recognize the meaning of words according to context, and generate the corresponding embedding for different contexts. For example, when generating the representation of the word \"_em be_\" (\"baby\") and the word \"_no_\" (\"it\") in the sentence \"_em be thich an kem vi no ngon_\" (\"the baby eats ice cream because it is delicious\"), context-free models generate different representations for the words \"_em be_\" and \"_no_\" because they are different words. In contrast, contextual embedding models learn that, in terms of meaning, the word \"_no_\" is the same as \"_em be_\". Therefore, contextual embedding models generate almost similar representations for the word \"_em be_\" and the word \"_no_\". In addition, the output features of the BERT model are fully exploited, and BERT generates a better representation than the context-free word-embedding model of NMT. Further, the pre-trained features of BERT were fused with NMT, enabling the NMT model to take advantage of BERT's features. Using the attention mechanism, this superior representation is then bridged to each layer of the encoder and decoder of the NMT model. As a result of clear input, the decoder of the NMT model generates better target sentences. Our experiment also showed that using the Vietnamese BERT model in combination with NMT leads to a significant improvement in the quality of the Vietnamese-Korean translation system by 1.41 BLEU points and 2.54 TER points. Furthermore, using BERT also improves the results of POS tagging, which is an annotation of the Vietnamese data.