# Text Data Augmentation for the Korean Language
## 2 Related Works
### Korean Corpora and Their Pre-Trained Language Models  
The training of large neural networks with the goal of language modeling has made significant improvements to NLP tasks. Typically, Bidirectional Encoder Representations from Transformers (BERT) [13] is a pre-trained deep bidirectional representation with a masked language model and the next sentence prediction objective. BERT is based on Transformer [16], an attention-based backbone that provides structured memory to handle long-term dependencies in a sequential structure. BERT uses cross-encoders to perform full attention on input pairs. On the other hand, the bi-encoder used in SentenceBERT [15] maps each input independently to a dense vector space. While cross-encoders typically achieve higher performance, they are extremely computationally expensive for many tasks. On the other hand, the bi-encoder is computationally efficient, but it often achieves lower performance than the BERT cross-encoder. Machine translation needs large parallel corpora of paired sentences in the source and target languages. However, bitext is limited, while a larger amount of monolingual data exists [12]. Study [3] presented a single attention-based seq2 seq NMT model to translate between multiple languages. Their solution did not require changing the model architecture from the standard NMT system but instead introduced a dummy token at the beginning of the input sentence to identify the required target language. Using a common lexicon between languages, their approach allows multilingual NMT to use a single model with significant performance gains, despite no increase in parameters. Furthermore, extension from the monolingual model to the multilingual model can be accomplished by knowledge distillation [17]. The training is based on the idea that a pair of source and target sentences must be mapped to the same location in vector space. The authors used the monolingual model as a teacher to generate embedded sentences for the source language and then train a new system as a student model on the translated sentences to mimic the teacher model. However, multilingual models demand a huge vocabulary, so they are often a burden on deriving tasks, such as fine-tuning on a particular language. Korean is often referred to in the research community as a low-resource language, due to the lack of resources and inadequate advertising and curation [18]. Compared to the industrial demand, the interest in Korean natural language processing has not yet received widespread attention from academic research. Korean is an agglutinative language, so there will be more problems in vocabulary expression because of its morphological diversity [19]. Korean tokenizes most words as single characters rather than morpheme-like units, and each token is unlikely to be separated by a \"space\" character, as in English. Furthermore, Korean consists of more than 10,000 syllabic characters, requiring both a vocabulary and an encoder to efficiently handle a variety of complex word forms. These characteristics limit the ability to derive downstream tasks from a pre-trained model in languages other than Korean itself. Several Korean language datasets have been reported in the recent literature to address the issue discussed here. Regarding natural language comprehension tasks, for example, [PERSON] et al. [20] introduced KorNLI and KorSTS for natural language inference (NLI) and sematic textual similarity, respectively. These two datasets were constructed using neural machine translation from English and partly by manual translation and re-labeling. Although there are a few datasets that come with scientific reports on them, most Ko rean language datasets are published scattered across the internet. To address this issue, [PERSON] et al. systematically described Korean corpora in a practical report [18]. The pre-trained language model for Korean is also incomplete, though a few publications providing a pre-trained model are available--for instance, KoreALBERT [21] and BERT [19].