# Exploring the Data Efficiency of Cross-Lingual Post-Training in Pretrained Language Models
## 5 Results and Discussion
### Extristic Evaluation  
The results after using all 4 M post-training examples are summarized in Table 3, alongside the results from fine-tuning publicly available models. All models are trained 10 times with different random seeds, and we report the mean and standard deviation. It can be seen that the proposed XPT outperforms Adapters and Scratch by a large margin across all tasks, demonstrating its effectiveness given the same amount of training data. Further, it also outperforms mBERT in all tasks and XLM-R in three out of four tasks. The fact that XPT is post-trained with only a fraction of the data used to train these massively multilingual language models makes this result more encouraging. When compared to KoBERT, a monolingual model trained from scratch with approximately 5.3 times more training examples, XPT still performs better in all tasks except KQP, with a 22.30% relative error reduction in KorSTS. Interestingly, the Scratch model outperforms all models in the KHS dataset. We believe that this is caused by a domain mismatch between the pre/post-training data and the fine-tuning data, as the KHS dataset is collected from social media. Figure 3: Intrinsic evaluation results as a function of the number of training examples. Similar to the intrinsic evaluations, we also experiment with varying the number of post-training sentences in the downstream evaluations and plot the results in Figure 4. From Figure 4,b, it can be seen that the Scratch setting shows minimal gains by increasing the data size from 100 K sentences to 800 K sentences. On the other hand, the models transferred from English show consistent improvements with more data. This shows that without any prior knowledge, these tasks cannot benefit from pretraining until a certain amount of data is available. We find a similar yet less pronounced trend in the KQP task as well. However, the results from the KHS task is different from the other three tasks, with all models performing comparably across all dataset sizes. Again, this is likely caused by the domain shift. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**Pre/Post-Train Data**} & \multicolumn{4}{c}{**Task**} \\ \cline{2-7}  & **Domain** & **\# Tokens** & **PAWS-X** & **KorSTS** & **KQP** & **KHS** \\ \hline mBERT & Wiki-100 & 6 B\({}^{*}\) & 81.37 \(\pm\) 0.63 & 77.48 \(\pm\) 0.73 & 92.30 \(\pm\) 0.70 & 61.66 \(\pm\) 1.56 \\ XLM-R & CC-100 & 295 B & **81.54**\(\pm\) 0.34 & **78.63**\(\pm\) 0.63 & 93.21 \(\pm\) 0.48 & 63.45 \(\pm\) 1.09 \\ KoBERT & Wiki/news-ko & 324 M & 79.68 \(\pm\) 0.75 & 77.67 \(\pm\) 2.86 & **93.54**\(\pm\) 0.75 & **64.68**\(\pm\) 0.64 \\ \hline Scratch & Wiki-ko & 61 M & 73.40 \(\pm\) 0.42 & 74.38 \(\pm\) 0.73 & 91.65 \(\pm\) 0.55 & **65.55**\(\pm\) 1.12 \\ Adapters & Wiki-ko & 61 M & 79.69 \(\pm\) 0.87 & 79.78 \(\pm\) 0.85 & 91.94 \(\pm\) 0.34 & 62.90 \(\pm\) 1.79 \\ XPT-SP (Ours) & Wiki-ko & 61 M & 80.46 \(\pm\) 0.50 & 80.98 \(\pm\) 0.40 & 92.22 \(\pm\) 0.46 & 64.22 \(\pm\) 1.24 \\ XPT (Ours) & Wiki-ko & 61 M & **81.62**\(\pm\) 0.38 & **82.65**\(\pm\) 0.39 & **92.88**\(\pm\) 0.29 & 64.85 \(\pm\) 0.53 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Downstream evaluation results. Best values in each block are highlighted in bold, and the overall bests are highlighted with underline. \({}^{*}\) Estimated value by extrapolation. Figure 4: Downstream evaluation results as a function of the number of pre/post-training examples. Investigating the change in data efficiency, it can be seen that to reach the performance of transferred models post-trained with 100 K examples, the Scratch model requires approximately 32, 16, and 16 times more pre-training data in the PAWS-X, KorSTS, and KQP tasks, respectively. Between the transferred models, XPT consistently outperforms Adapters in PAWS-X, KorSTS, and KHS, while performing on par in KQP. This indicates that regardless of the amount of available post-training data, XPT can be expected to perform better than Adapters.