# Exploring the Data Efficiency of Cross-Lingual Post-Training in Pretrained Language Models
## 6 Conclusions and Future Directions  
While being highly effective across a wide range of NLP tasks, pretrained Transformers are extremely data-hungry. For the majority of the over 7000 languages spoken worldwide, it is difficult to secure sufficient data for training such models. In this paper, we tackled this problem by proposing an approach for data-efficient training of pretrained language models in a low-resource language. Our approach, termed XPT, achieves this goal by post-training a PLM from another high-resource language. Language-agnostic parameters of a model trained in the high-resource language are selectively reused and tuned while learning the language-specific parameters in the target language. We also proposed ITL, which is designed to learn linguistic differences between the two languages at a sequence level instead of a token level. To evaluate our method in a challenging and controlled scenario, we conducted a case study for Korean by post-training English RoBERTa with a varying amount of post-training examples. Intrinsic results have shown that post-training an English model in \(L_{T}\) is roughly as effective as using twice as much data in \(L_{T}\) and training from scratch. Further, downstream evaluations on four natural language understanding tasks demonstrated that our approach can improve the data efficiency by a factor of up to 32. When compared to monolingual and massively multilingual PLMs trained with several orders of magnitude more data, XPT still outperformed or matched the performance. This suggests that completely adapting a model to a single language of interest is more effective and efficient, and more focus should be put on this direction of research. As for future directions, experimenting with other target languages with different resource availability and linguistic characteristics is an important step. Building a systematic approach for selecting the source language depending on the target language is also a promising research direction. Conceptualization, C.L. and T.W. ; data curation, C.P. ; formal analysis, C.L. and A.M.; funding acquisition/project administration/supervision, H.L. ; investigation, C.L. ; methodology/software, K.Y. and C.P. ; review and editing, T.W. and A.M.; original draft, C.L. and K.Y. All authors have read and agreed to the published version of the manuscript. **Funding:** This work was supported by Institute for Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2020-0-00368, A Neural-Symbolic Model for Knowledge Acquisition and Inference Techniques). Additionally, it was also supported by the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2020-2018-0-01405) supervised by the IITP (Institute for Information and Communications Technology Planning and Evaluation). **Institutional Review Board Statement:** Not applicable. **Informed Consent Statement:** Not applicable. **Data Availability Statement:** Publicly available datasets were analyzed in this study. The data can be found here: PAWS-X: [[https://github.com/google-research-datasets/paws/tree/master/pawsx](https://github.com/google-research-datasets/paws/tree/master/pawsx)]([https://github.com/google-research-datasets/paws/tree/master/pawsx](https://github.com/google-research-datasets/paws/tree/master/pawsx)) (accessed on 22 February 2021). KorSTS: [[https://github.com/kakaobrain/KorNLUDatasets/tree/master/KorSTS](https://github.com/kakaobrain/KorNLUDatasets/tree/master/KorSTS)]([https://github.com/kakaobrain/KorNLUDatasets/tree/master/KorSTS](https://github.com/kakaobrain/KorNLUDatasets/tree/master/KorSTS)) (accessed on 22 February 2021). KQP: [[https://github.com/songys/Question_pair](https://github.com/songys/Question_pair)]([https://github.com/songys/Question_pair](https://github.com/songys/Question_pair)) (accessed on 22 February 2021).