# Text Data Augmentation for the Korean Language
## 4 Experiments and Results
### Downstream Task Evaluation  
The evaluation process was common to all tasks: We repeated the trials five times on five different random pre-sets, as depicted above. In each experiment, we applied text data augmentation on the pre-sets to generate the corresponding synthetic sets, then took the union of the pre-set and the synthetic set to build the augmented set. We applied cross-validation with a 20% portion as a validation set from the augmented set for fine-tuning. Finally, we assessed the performance of the fine-tuned model on the original test set of each task, and we report the average outcome across the five different experiments, as shown in Tables 3 and 4. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multirow{2}{*}{
\begin{tabular}{c} **Full Training** \\ **Dataset** \\ \end{tabular} } & \multirow{2}{*}{**Pre-Dataset**} & \multicolumn{2}{c}{**Augmented Dataset**} \\ \cline{3-5}  & & & **EDA** & **Back Translation** \\ \hline KorNLI & 82.24 & 71.25 & **73.49** & 72.13 \\ \hline Question Pair & 95.25 & 94.19 & **94.85** & 93.13 \\ \hline KorSTS & 85.53 & 81.67 & 82.54 & **83.86** \\ \hline NSCM & 90.63 & 86.32 & **86.40** & 85.60 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Performance on downstream tasks of pre-trained KoELECTRA-Base-v3: NLI, natural language inference (accuracy); QUAD, question duplication (F1); STS, semantic textual similarity (Spearman); NSCM, sentiment classification (accuracy). \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multirow{2}{*}{
\begin{tabular}{c} **Full Training** \\ **Dataset** \\ \end{tabular} } & \multirow{2}{*}{**Pre-Dataset**} & \multicolumn{2}{c}{**Augmented Dataset**} \\ \cline{3-5}  & & & **EDA** & **Back Translation** \\ \hline KorNLI & 83.21 & 71.27 & **73.93** & 72.81 \\ \hline KorSTS & 84.75 & 81.70 & 81.54 & **81.99** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Performance on downstream tasks of pre-trained ENILPLE-v2: NLI, natural language inference (accuracy); STS, semantic textual similarity (Spearman). For the learning model, we utilized two Korean-monolingual pre-trained models from two different resources: ENLIPLE-v2 and ELECTRA-Base-v3. The pretrained weights were originally obtained from the official release of each model. Hence, we only report the settings used in this study. We by-turn fine-tuned each learning model on the dataset of each specific task using the cross-entropy objective function, on 10 epochs with the Adam optimizer. The batch size was set to 32, and sentences with lengths less than 128 were padded with \"[PAD] = 0\" at their ends. The learning rate was warmed up within two epochs and kept unchanged at \(5\times 10^{-5}\). To obtain the results shown in Tables 3 and 4, we generated synthetic data from a random pre-set using EDA and back translation. We created a new instance for each sample in the pre-set, which means that the number of samples generated was equal to the number of elements of the pre-set. For EDA settings, we randomly picked one of four transformations (SR, RI, RS, RD) with a replacement probability of 20%. In terms of implementation, we referred to the original implementation of EDA for English and Korean versions ([[https://github.com/toriving/KoEDA](https://github.com/toriving/KoEDA)]([https://github.com/toriving/KoEDA](https://github.com/toriving/KoEDA)) (21 February 2022)). In this way, our augmented set was twice the size of the pre-set. We also note that the EDA method runs remarkably faster than back translation: under 10 ms with EDA, compared to 1 s with back-translation. For pre-trained models, we used ENLIPLE-v2 (results shown in Table 3) and KoELECTRA-Base-v3 (results shown in Table 4). ENLIPLE-v2 is the pre-trained language model based on dynamic masking from RoBERTa [23] and n-gram masking from AlBERT [24]. The model was trained on Korean Wikipedia and news in a total of 174M sentences, with mask language modeling (15% masking), next sentence prediction, and sentence order prediction objective.