# Exploring the Data Efficiency of Cross-Lingual Post-Training in Pretrained Language Models
## 5 Results and Discussion
### Intrinsic Evaluation  
To quantitatively measure the learning process of each model, we first perform intrinsic evaluations. While performing better on intrinsic tests does not guarantee the model to be better at downstream tasks as well, they are good estimates and relatively cheap to evaluate. In this study, perplexity, hits@k, mean rank, and mean reciprocal rank are calculated on the masked tokens and used as intrinsic metrics.  
The intrinsic performance of each model after using all available 4M training examples is summarized in Table 2. It can be seen that XPT performs the best, followed by Adapters and Scratch. Further, the improvement from Scratch to Adapters is as large as the improvement from Adapters to XPT, indicating that our proposed XPT performs much favorably to Adapters.  
To understand the data efficiency during the post-training process, we exponentially vary the number of training examples from 100 K to 3200 K. The result for each metric is plotted in Figure 3. As can be seen, the performance increases linearly as the dataset size increases exponentially. This shows how data-hungry these language models are, demonstrating the importance of increasing the data efficiency. Adapters and XPT perform comparably, with XPT performing slightly better when there are more than 400 K training examples and slightly worse with less than 400 K examples.  
On the other hand, we find that transferring from English (i.e., Adapters and XPT) is roughly as effective as doubling the number of training examples in the Scratch setting. Further, the difference between transferred and non-transferred settings are most pronounced when the amount of data is minimal, suggesting that low-resource languages are likely to benefit the most from XPT. The common underlying hypothesis in the multilingual language modeling literature is that simultaneously learning from multiple languages is the key to improving performance in low-resource languages by learning polyglot representation [12; 13; 36; 37; 38]. However, our results suggests that some part of the knowledge acquired by monolingual models is language-agnostic and thus can be effectively transferred to other languages. Based on this, we argue that more emphasis should be put on transferring monolingual representation [44; 45] as these are more sustainable than multilingual training.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Model** & **PPL** & **Hits@1** & **Hits@3** & **Hits@5** & **Hits@10** & **MRR** & **MRR** \\ \hline Scratch & 5.88 & 63.91 & 76.18 & 80.41 & 85.18 & 0.7145 & 58.87 \\ Adapters & 5.46 & 65.05 & 77.28 & 81.40 & 86.04 & 0.7253 & 53.93 \\ XPT-SP (Ours) & 5.41 & 65.28 & 77.49 & 81.60 & 86.18 & 0.7273 & 54.03 \\ XPT (Ours) & **5.07** & **66.13** & **78.31** & **82.38** & **86.90** & **0.7354** & **50.41** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Intrinsic evaluation results after using all available 4 million training examples in \(L_{2}\). Best values are highlighted in bold. PPL = perplexity. MRR = mean reciprocal rank. MR = mean rank.