SlowGuess/NAACL_2022_OCR huggingface dataset:
RoBERTa models with existing comparable models trained from scratch with more training effort. We consider the total number of tokens the model has encountered in the target language, computed as the product of batch size  $\times$  sequence length  $\times$  train steps (shown in Table 1) as a proxy for training effort. We evaluate the performance of CamemBERT (Martin et al., 2020) (French),  $\mathrm{GBERT_{Base}}$  (Chan et al., 2020) (German), and  $\mathrm{BERT_{Base}-Chinese}$  (Devlin et al., 2019) as existing monolingual LMs, as well as XLM- $\mathbf{R}_{\mathrm{Base}}$  (Artetxe et al., 2020) as a high-performing multilingual LM.
We present our results on transferring RoBERTa and GPT-2 from English to other languages, followed by analyzing training behavior. In Appendix C, we provide a qualitative assessment of how well subword tokens are mapped between the source and the target languages.
5.1 Transferring RoBERTa
Table 2 reports the evaluation results of RoBERTa. As shown, models initialized with WECHSEL outperform models trained from scratch and models initialized with TransInner across all languages.
Surprisingly, close relatedness of the source and target language is not necessary to achieve effective transfer, as e.g. on NLI WECHSEL improves absolute accuracy by  $7.15\%$ ,  $6.31\%$ ,  $6.94\%$  and  $4.71\%$  over models trained from scratch for French, German, Chinese and Swahili, respectively.
We observe that our parameter transfer-based model consistently outperforms the previously released LMs on both monolingual and multilingual settings, while these models benefit from much larger training resources in terms of computation time and corpus size. In particular, the results show an improvement over XLM-R<sub>Base</sub> by an average  $3.54\%$  accuracy for NLI and  $1.14\%$  micro F1 score for NER. For NLI, we improve over the prior monolingual models by  $1.55\%$ ,  $3.15\%$  and  $1.77\%$  absolute accuracy for French, German and Chinese, respectively. For NER, we observe improvements over monolingual models with  $0.62\%$  and  $0.26\%$  absolute micro F1 score improvement for French and German, respectively. For Chinese, the monolingual model BERT<sub>Base</sub>-Chinese still outperforms our method by  $1.5\%$  absolute micro F1 score. We suspect that the discrepancy between NLI and NER is due to the limited training corpus size (max. 4GiB), while a larger corpus can potentially improve NER as more named entities appear (Martin et al., 2020).
To the best of our knowledge there is no monolingual model available for Swahili.
Figure 2: Test scores over training steps from fine-tuning RoBERTa models on NLI (using XNLI) and NER (using WikiANN). Perplexity on the held-out set over training steps of GPT-2 models. We evaluate every 12.5k steps.
Lang	Model	PPL@0	PPL@25k	PPL@250k
French	WECHSEL-GPT2	1.7e+3	23.47	19.71
TransInner-GPT2	1.4e+5	67.97	20.13
FullRand-GPT2	5.9e+4	25.99	20.47
German	WECHSEL-GPT2	3.7e+3	34.35	26.80
TransInner-GPT2	1.5e+5	121.67	27.76
FullRand-GPT2	5.8e+4	37.29	27.63
Chinese	WECHSEL-GPT2	2.4e+4	71.02	51.97
TransInner-GPT2	1.5e+5	231.05	56.17
FullRand-GPT2	5.8e+4	69.29	52.98
Swahili	WECHSEL-GPT2	1.4e+5	13.02	10.14
TransInner-GPT2	1.4e+5	42.95	10.28
FullRand-GPT2	5.8e+4	13.22	10.58
Table 3: Results of training GPT2 models. We report Perplexity before training (PPL@0), after  $10\%$  of steps (PPL@25k) and after training (PPL@250k).
Lang	Model	Best PPL
Sundanese	WECHSEL-GPT2	111.72
TransInner-GPT2	151.86
FullRand-GPT2	149.46
Scottish Gaelic	WECHSEL-GPT2	16.43
TransInner-GPT2	18.62
FullRand-GPT2	19.53
Uyghur	WECHSEL-GPT2	34.33
TransInner-GPT2	39.06
FullRand-GPT2	42.82
Malagasy	WECHSEL-GPT2	14.01
TransInner-GPT2	14.85
FullRand-GPT2	15.93
Table 4: Results of training GPT2 models on low-resource languages. We report the best Perplexity on the held-out set, evaluated every  $2.5\mathrm{k}$  steps. See Figure 3 for Perplexity throughout training.
The first two columns of Figure 2 show the performance of RoBERTa models on downstream tasks after each  $12.5\mathrm{k}$  training steps. Models initialized with WECHSEL reach high performance in significantly fewer steps than models initialized with FullRand or TransInner.
We expect FullRand-RoBERTa to approach performance of the respective prior monolingual models when trained on the same amount of tokens.
It would presumably be slightly worse because we restrict training corpus size to 4GiB.
For French, WECHSEL-RoBERTa outperforms CamemBERT after  $10\%$  of training steps, reducing training effort by 64x. For German, WECHSEL-RoBERTa outperforms  $\mathrm{GBERT_{Base}}$  after  $10\%$  of training steps, reducing training effort by 39x. For Chinese, WECHSEL-RoBERTa outperforms  $\mathrm{BERT_{Base}}$ -Chinese on NLI, but does not outperform  $\mathrm{BERT_{Base}}$ -Chinese on NER.
5.2 Transferring GPT-2
5.2.1 To Medium-Resource Languages
Results on medium-resource languages are shown in Table 3. Similar to the results for WECHSEL-RoBERTa, the GPT-2 models trained with WECHSEL consistently outperform the models trained from scratch and with TransInner across all languages.
The rightmost column of Figure 2 depicts the performance of GPT-2 models after each  $12.5\mathrm{k}$  training steps. Comparing the results across all languages throughout training, we observe a stronger dependence on similarity of the source to the target language than for downstream tasks such as NLI or NER. In particular, for French and German, WECHSEL is consistently better than TransInner and FullRand throughout the entire training, while for Chinese, a decrease in perplexity towards the end of training causes WECHSEL to surpass training from scratch.
5.2.2 To Low-Resource Languages
Table 4 reports the perplexity of Language Modelling on the low-resource languages. Again, we observe consistent improvements using WECHSEL on all languages. Furthermore, we find that the improvement from WECHSEL tends to increase as the amount of training data decreases by conducting a sensitivity analysis w.r.t. the amount of available training data (Appendix F).
In Figure 3 we report the performance of the low-resource LMs on the held-out set throughout training. One difference of the low-resource models with the ones trained on medium-resource languages is that the low-resource LMs are prone to overfitting, and require appropriate model selection even in the early steps of training. Notably, TransInner-GPT2 takes more steps to overfit since all non-embedding parameters are frozen for the first 25k steps (c.f. Section 4).
Figure 3: Perplexity throughout training on low-resource languages. We evaluate every 2.5k steps and stop training if Perplexity on the held-out set does not improve for 10k steps.
WECHSEL-GPT2
Translner-GPT2
FullRand-GPT2
5.3 Is freezing necessary?
Previous work using the TransInner method freezes non-embedding parameters for a fixed amount of steps before training the entire model (de Vries and Nissim, 2021). This is done to prevent catastrophic forgetting at the beginning of training. To evaluate if freezing non-embedding parameters is still necessary with our method, we conduct an additional experiment. We train a German GPT-2 model with WECHSEL and a model with TransInner without freezing any parameters, and the same models with freezing of non-embedding parameters for the first  $10\%$  of steps. We match hyperparameters of the main experiments except training for  $75\mathrm{k}$  steps only. Based on the results shown in Figure 4, we conclude that freezing is necessary when using TransInner, but there is no need for freezing when using WECHSEL.
Figure 4: Comparison of German GPT-2 models trained with WECHSEL and TransInner between freezing non-embedding parameters at the start and not freezing any parameters.
We conduct our experiments on up to eight languages, showing the benefits of our parameter transfer method to both medium- and low-resource languages. However, there are many more languages with diverse linguistic characteristics on which our WECHSEL method is not tested. This is a limitation forced by computational constraints, as we can not ascertain whether transfer to all other languages would result in similar improvements. In addition, our extrinsic evaluation is limited to two tasks (NLI and NER). While this choice is due to the limitations on the available collections in various languages, this evaluation does not necessarily provide a comprehensive view of language understanding tasks.
6.2 Risks
It is well-known that existing LMs trained on English text encode societal biases (Bolukbasi et al., 2016; Caliskan et al., 2017; Rekabsaz et al., 2021b) and stereotypes and using them in downstream tasks might lead to unfair treatment of various social groups (Zerveas et al., 2022; Krieg et al., 2022; Ganhor et al., 2022; Rekabsaz et al., 2021a; Melchiorre et al., 2021; Rekabsaz and Schedl, 2020; Elazar and Goldberg, 2018). Since we propose a method to transfer the English LMs to new languages, it is highly probable that the existing biases are also transferred to the target LMs. We therefore advocate a conscious and responsible use of the transferred LMs in practice.
We introduce WECHSEL, an effective method to transfer monolingual language models to new languages. WECHSEL exploits multilingual static word embeddings to compute an effective initialization of subword embeddings in the target language. We conduct experiments by transferring RoBERTa and GPT-2 models from English to French, German, Chinese and Swahili, as well as English GPT-2 to four low-resource languages. The evaluation results show that the transferred RoBERTa and GPT-2 models are more efficient and effective than strong baselines, and consistently outperform prior monolingual models that have been trained for a significantly longer time. WECHSEL facilitates the creation of effective monolingual LMs for new languages with medium to low resources, particularly in computationally-limited settings. In addition, our work provides strong evidence towards the hypothesis by Artetxe et al. (2020) that deep monolingual language models learn abstractions that generalize across languages.
Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). We thank Andy Koh and Artus Krohn-Grimberghe for providing additional computational resources. The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are supported by the Federal State Upper Austria. We thank the project INCONTROL-RL (FFG-881064). Research also supported in part by the NSF (IIS-1956221), the State of Upper Austria and the Austria's Federal Ministry of Education, Science, and Research through the project FAIRFLOW (LIT-2021-YOU-215).
Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9-15, Marseille, France. European Language Resource Association.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2289-2294, Austin, Texas. Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451-462, Vancouver, Canada. Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789–798, Melbourne, Australia. Association for Computational Linguistics.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623-4637, Online. Association for Computational Linguistics.
Piotr Bański and Beata Wojtowicz. 2009. Freedict: an open source repository of tei-encoded bilingual dictionaries.
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610-623.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proc. of NeurIPS.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science.
Branden Chan, Stefan Schweter, and Timo Möller. 2020. German's next language model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6788-6796, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishray Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. arXiv preprint arXiv:1710.04087.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating crosslingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Wietse de Vries and Malvina Nissim. 2021. As good as new. how to successfully recycle English GPT-2 to make models for other languages. In Findings of the Association for Computational Linguistics: ACLIJCNLP 2021, pages 836-846, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Long Duong, Hiroshi Kanayama, Tengfei Ma, Steven Bird, and Trevor Cohn. 2016. Learning crosslingual word embeddings without bilingual corpora. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1285-1295, Austin, Texas. Association for Computational Linguistics.
Yanai Elazar and Yoav Goldberg. 2018. Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961.
Christian Ganhör, David Penz, Navig Rekabsaz, Oleg Lesota, and Markus Schedl. 2022. Mitigating consumer biases in recommendations with adversarial training. In Proceedings of the 45th International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2022. ACM.
Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. 2018. Loss in translation: Learning bilingual word mapping with a retrieval criterion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2979-2984, Brussels, Belgium. Association for Computational Linguistics.
Klara Krieg, Emilia Parada-Cabaleiro, Markus Schedl, and Navid Rekabsaz. 2022. Do perceived gender biases in retrieval results affect relevance judgements? In Proceedings of the Workshop on Algorithmic Bias in Search and Recommendation at the European Conference on Information Retrieval (ECIR-BIAS 2022).
Guillaume Lample, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In International Conference on Learning Representations.
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020. Pre-training via paraphrasing. In Advances in Neural Information Processing Systems, volume 33, pages 18470-18481. Curran Associates, Inc.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Antoine Louis. 2020. BelGPT-2: a GPT-2 model pretrained on French corpora. https://github.com/antoiloui/belgpt2.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Bilingual word representations with monolingual quality in mind. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 151-159, Denver, Colorado. Association for Computational Linguistics.
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoit Sagot. 2020. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203-7219, Online. Association for Computational Linguistics.
Alessandro B. Melchiorre, Navid Rekabsaz, Emilia Parada-Cabaleiro, Stefan Brandl, Oleg Lesota, and Markus Schedl. 2021. Investigating gender fairness of recommendation algorithms in the music domain. Information Processing and Management, 58(5):102666.
Toan Q. Nguyen and David Chiang. 2017. Transfer learning across low-resource, related languages for neural machine translation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers),
pages 296-301, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Debora Nozza, Federico Bianchi, and Dirk Hovy. 2020. What the [mask]? making sense of language-specific bert models. arXiv preprint arXiv:2003.02912.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Crosslingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946-1958, Vancouver, Canada. Association for Computational Linguistics.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001, Florence, Italy. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for NER. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151-164, Florence, Italy. Association for Computational Linguistics.
Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, and Omer Levy. 2021. Few-shot question answering by pretraining span selection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3066-3079, Online. Association for Computational Linguistics.
Navid Rekabsaz, Simone Kopeinik, and Markus Schedl. 2021a. Societal biases in retrieved contents: Measurement framework and adversarial mitigation of bert rankers. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, page 306-316.
Navid Rekabsaz, Nikolaos Pappas, James Henderson, Banriskhem K Khonglah, and Srikanth Madikeri. 2019. Regularization advantages of multilingual neural language models for low resource domains. arXiv preprint arXiv:1906.01496.
Navid Rekabsaz and Markus Schedl. 2020. Do neural ranking models intensify gender bias? In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2065-2068.
Navid Rekabsaz, Robert West, James Henderson, and Allan Hanbury. 2021b. Measuring societal biases from text corpora with smoothed first-order co-occurrence. In Proceedings of the Fifteenth International AAAI Conference on Web and Social Media, ICWSM 2021, held virtually, June 7-10, 2021, pages 549-560. AAAI Press.
Phillip Rust, Jonas Pfeiffer, Ivan Vulic, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3118-3135, Online. Association for Computational Linguistics.
Peter H Schonemann. 1966. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1-10.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy. Association for Computational Linguistics.
Ke Tran. 2020. From english to foreign languages: Transferring pre-trained language models. arXiv preprint arXiv:2002.07306.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120-130, Online. Association for Computational Linguistics.
Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1006-1011, Denver, Colorado. Association for Computational Linguistics.