# Text Data Augmentation for the Korean Language
## 3 Data Augmentation
### Back Translation  
Back translation is an unsupervised approach for text data augmentation [2; 12] that refers to the scheme of translating an example \(\mathbf{x_{A}}\) into an example \(\mathbf{x_{B}}\) in another language, and then translating it back to obtain an augmented example \(\hat{\mathbf{x}_{A}}\). As an open vocabulary modeling approach, back translation can generate diverse paraphrases while preserving the semantics of the original sentences, leading to significant performance improvements in semantic reasoning. Supposed that we have a machine translation model from Language A (e.g., Korean), \(p_{A\to B}(\mathbf{x_{B}}|\mathbf{x_{A}})\), that translates a sentence \(x_{A}\) into \(x_{B}\), and we also have a reverse translation model from \(\mathbf{x_{B}}\) into \(\hat{\mathbf{x}_{A}},\ p_{B\to A}\ (\hat{\mathbf{x}_{A}}\ |\mathbf{x_{B}})\). We define our data augmentation generative model as \(p_{\theta}\left(x_{aug}|x_{pre}\right)=\sum_{x_{B}\in TS}p_{B\to A}\ (\ x_{aug}\ |\mathbf{x_{B}})p_{A\to B}(\mathbf{x_{B}}|\mathbf{x_{pre}})\), where \(x_{B}\in TS\) is all possible translated versions in English of \(\mathbf{x_{pre}}\) in Korean, and \(\mathbf{x_{aug}}\) is a synthetic example in Korean generated by back translation from \(\mathbf{x_{B}}\), while \(\theta\) is parameterized by pre-trained weights from both translation models.  
Although back translation is not likely to mislabel the synthetic example, it suffers other weaknesses, such as out-of-vocabulary terms. For word-level NMT models, the translation of non-existing words has been solved through backup dictionary lookups [20; 21]. However, such techniques are impractical, since because of differences in the morphological synthesis between languages, there is no one-to-one mapping between the source and target vocabulary. In this study, we overcome the out-of-vocabulary problem by using apre-trained model on a large dictionary. In particular, we utilize multilingual translation based on mBART [22]. The set of pre-trained weights ([[https://github.com/UKPLab/EasyNMT](https://github.com/UKPLab/EasyNMT)]([https://github.com/UKPLab/EasyNMT](https://github.com/UKPLab/EasyNMT)) (20 February 2022)) is fine-tuned from 50 languages, which comprises a parallel corpus ([[https://sites.google.com/site/iwsltevaluation2017/TED-tasks](https://sites.google.com/site/iwsltevaluation2017/TED-tasks)]([https://sites.google.com/site/iwsltevaluation2017/TED-tasks](https://sites.google.com/site/iwsltevaluation2017/TED-tasks))) between Korean and English.  
Table 2 shows examples of the two approaches. In this example, the SR approach replaces the subject of the sentence \"bicycle\" with \"cycle\" and the subject \"man\" with \"child\", but does not change the context of the action \"is riding\". In RL, the first example shows the insertion of \"local\" to support the subject \"man\", but the second example does not make sense when adding \"range\" to the original sentence. As shown in the case of RD and RS, swapping and deleting words make sentences absurd and grammatically incorrect. Interestingly, in the case of BT, the context of the sentence remains unchanged, even though the grammar is legitimately changed.