Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studies
Authors: Zhengxuan Wu
Alex Tamkin
Isabel Papadimitriou
Venue: EMNLP 2023 Main (2023)
Keywords: crosslingual
knowledge transfer
language model
finetuning
TL;DR: We identify factors most important in crosslingual transfer by performing controlled transfer experiments between pretrained English models and transformed Englishes that vary on just one axis of variation.

Abstract: When we transfer a pretrained language model to a new language, there are many axes of variation that change at once. To disentangle the impact of different factors like syntactic similarity and vocabulary similarity, we propose a set of \emph{controlled transfer studies}: we systematically transform the language of the GLUE benchmark, altering one axis of crosslingual variation at a time, and then measure the resulting drops in a pretrained model's downstream performance. We find that models can largely recover from syntactic-style shifts, but cannot recover from vocabulary misalignment and embedding matrix re-initialization, even with continued pretraining on 15 million tokens.
Moreover, good-quality tokenizers in the transfer language do not make vocabulary alignment easier. Our experiments provide insights into the factors of cross-lingual transfer that researchers should most focus on when designing language transfer scenarios.

Oolong: Investigating What Makes
Transfer Learning Hard with Controlled Studies
Zhengxuan Wu", Alex Tamkin', Isabel Papadimitrion"
Stanford University (wuzhengx, atamkin, isabelvpy@stanford.
Abstract
finetune
GLUE
Score: 81%
When we transfer a pretrained language model a new language, there are many axes of vari- ation that change at once. To disentangle the impact of different factors like syntactic simi- larity and vocabulary similarity, we propose set of controlled transfer studies; we system- atically transform the language of the GLUE benchmark, altering axis of crosslingual variation at a time, and then measure the result- ing drops pretrained model's downstream performance. We find that models can largely recover from syntactic-style shifts, but can- recover from vocabulary misalignment and embedding matrix re- re-initialization, even with continued pretraining on 15 million to- kens. Moreover, good-quality tokenizers in the transfer language do not make vocabulary align- ment easier. Our experiments provide insights into the factors of cross- poosullnguaa transfer that re- searchers should most focus on when designing language transfer scenarios.
LM
transform
GLUE
GLUE
[syatas, tokesiasr,
emboaMinggs)
finetune
GLUE
Score: ? %
LM
Figure 1: Controlled transfer studies paradigm. We systematically transform GLUE tasks (t-GLUE) to tar- get linguistic factor, then finetune a pretrained lan- guage model that dataset. The resulting drop in performance indicates the importance of that factor to crosslingual transfer. See Table for the list of transfor- mations.
successful transfer, we present set of controlled transfer studies to compare the effects of different aspects cross-lingual shift.
Our controlled studies consist of transferring an English model to a language that is transformed from English on just one axis of variation. Realis- tic transfer scenarios involve languages that differ across multiple axes of variation at time. Our experiments serve disentangle these effects, and identify the issues that practitioners should most focus on when doing cross-lingual transfer learn- ing. We examine three factors that are salient in a transfer learning context:
1 Introduction
What makes it hard for neural networks to learn new languages? Large language models (LLMs) require vast datasets for pretraining, making challenging to train LLMs from scratch for low- resource languages (Devlin et al., 2018; Liu et al., 2019; Lacoste et al., 2019; Clark et al., 2020). For such languages, appealing approach is to trans- fer knowledge from LLM trained for a high- resource language, especially since pretrained mod- els can transfer knowledge across even extreme shifts (Papadimitriou and Jurafsky, 2020; Tamkin et al., 2020), A range of methods have been ex- plored to enable such crosslingual transfer of En- glish LLMs, using techniques such as adaptive pretraining (Reimers and Gurevych, 2020), and embedding retraining (Artetxe et al., 2020; Tran, 2020). To better understand the factors affecting
Word-order syntactic differences: Lan- guages vary greatly in the ways that their syn- tax orders words. Syntactic topological simi- laritics are generally considered important factor when deciding transfer language pairs. We the effects different levels word- order perturbation in transfer learning.
Word identity alignments: Transferring to a new language requires learning the meaning, or word embeddings, of new words, and how
In Chinese, "Oolong" can refer to an unexpected change or development. 'Equal contribution. *Coresponding author.

---

their layer embeddings correspond the old language. We experiment with the effect of re- initializing or shuffling the rows of the layer word embedding matrix before transfer.
experiments build off previous efforts that try en- able crosslingual transfer from pretrained monolin- gual LLMs to new languages (Artetxe et al., 2018, 2020; Tran, 2020; Reimers and Gurevych, 2020; Gogoulou et al., 2021).
Tokenizer quality We test the effect of bad tokenizer quality by reinitializing the word embedding matrix and transferring to English data tokenized with French and Dutch tok- enizers that are suboptimal quality for English tokenization.
With respect to vocabulary sharing and adap- tation, Liang et al. (2023) show that training a multilingual model with a massive vocabulary that separates out languages outweighs the benefits of vocabulary sharing between language (Patil et al., 2022), while in the transfer regime Chronopoulou et al. (2020) showcase the importance of main- taining vocabulary overlap. Techniques mapping subword embeddings to their new synonyms, or keeping subwords in the same script across lan- guages. prove effective for cross-lingual transfer (Vernikos and Popescu-Belis, 2021; Pfeiffer et al., 2021, 2020; Muller et al., 2021). The importance of embedding intialization statistics is discussed in (Raghu et al., 2019).
We test the effect of these factors on transfer learning both by 1) directly fine-tuning on t-English versions of the GLUE benchmark, as well as 2) continuing masked language model pre-training on 15 million tokens of t-English wikitext. In all cases, we find that word identity alignment provides the greatest stumbling block for transfer learning. Re- initializing or shuffling the rows of the embedding matrix has a very negative effect on downstream learning which we cannot reverse in the low-data simulating. If the embedding
Results on the importance of syntactic shifts main broad, with work on multilingual training suggesting that syntactic shifts are significant com- pared to vocabulary effects (K et al., 2020), and that syntactic structure plays a role in developing par- allel multilingual encodings (Dufter and Schütze, 2020), while Deshpande et al. (2022) show inter- secting effects of vocabulary and word order shifts. Understanding the direct relationship between the effect of syntactic shifts and the effect of vo- cabulary and tokenizer shifts remains an important problem in understanding transfer learning. Our work creates a framework for decomposing and dis- entangling the difficulties of transfer in controlled studies, giving researchers pointers for what pects of language variation make transfer difficult.
regime that matrix is reinitialized and a new tokenizer is used, the effect of reinitialization overshadows any effect that the quality of the new tokenizer might cause. In the case of syntactic word-order transformations, we find that even in the low-data transfer learning regime, the models we test can adapt to word order shifts as long as vocabulary information is kept.
We run experiments on RoBERTa, DeBERTa, and XLM-R in order to test transfer learning be- yond the training set languages for both monolin- gual and multilingual models. Our method allows us to disentangle the effects of correlated factors by inspecting them one at a time.!
2 Related Work
Methods
As self-supervised pretraining advances the state of NLP in high-resource languages, research into widening these successes beyond high-resource languages has become widespread and important. Methodologies for best transferring monolingual multilingual model to an unseen language are widely explored. Ogueji et al. (2021) and Ogun- remi et al. (2023), showcase the positive effects of pretraining on closer and related languages to the target language, even if this less data than larger pretrained models, in part because of the possibility of shared vocabulary (Oladipo et al., 2022). Our
Our methodology consists of taking pretrained model, and transferring to a t-English: a system- atically transformed version of English data that differs from English on one axis of variation. The different t-Englishes that we use are described and motivated below, and examples are in Table 1. We consider two low-data transfer environments: Di- rect Fine-tuning, where transfer the English pretrained model directly to t-GLUE, transformed GLUE datasets (Wang al., 2018), and Contin- ued Pretraining, where we first do masked lan- guage modeling training on 15 million tokens of the WikiText- 103M corpus (Merity et al., 2016)
'Our code is available publicly at https: tttt:::///ttubb.... com/ frankaging/oole oolong- crosslingual.

---

Transformation Type
Sentence/ / Sequeecce
Original Fnglish
"the film unfolds with all the mounting tension of an expert thriller, until the tragedy heneath it all gradually reveals itseff."
Random Order
"an all all gradually beneath thriller with reveals until tonsion tragedy mounting the of the the expert. unfolds itself film" inself reveals gradually all it beneath tragedy the unti), thriller expert an of tension mounting the all with unfolds the"
Reverse Order
(N.. Vu. (Nz, Via]
"the fibu with all the an expest. until the beneath all gradually . itself roveals tragedy thiiler tension mounting unfoids"
"the film unfolds with all the tension of an thriller, until the tragdyy beneath it all gradually itself. reveals expert moceting"
[Ne. Vyd
"the film unfolds with all the of an expert. until the beneath all gradually. itsell reveals it tragedy thriller tension mounting"
RoBERTa Tokenizer
"the film unfolds with all the mounting tension of an expen thriller, until the rmagedy heneath it all gradually reveals isself.
Tokenizer
"the film un fold with all the mounting tension expert thriller, until the tragedy beneuth
graduuily reveals itvelf. "
Albert Tokenizer
"the film unfold with all the mounting tension of expert thriller, until the tragedy heneath it all gradually reveals itsalf
FlauBERT Tokenizer
"the film un fol ds with all the mou ting tension of expest thrifler, un til the age dy bene ath all gradu ally re als it self
DutchBERT Tokenizer
"the film u s'old' s with all the mo unt ing te a sion of a a expert thriller, a til the trage d ben e ath i all gra d u ally e als il f."
Table An example from the SST-2 dataset and its t-English variants. Tokenizer pre-fixes and post-fixes such as G. ## and (/w) are not shown for simplicity.
transformed to t-English. 2
these gains may due several factors, including better initialization statistics (Raghu et al., 2019), or to a learned alignment between the learned em- beddings and the pretrained transformer layers (Wu et al., 2021). Here, we test the baseline effect of reinitializing the embedding layer while transfer- ring to the same language that the model was pre- trained. We compare this to a scenario where the rows of the embedding matrix are shuffled, mean- ing that vector statistics are broadly similar but each word has been swapped with another and the model needs to find the mapping during fine-tuning.
3.1 Transformed English (t-Englishes)
Syntactic Shifts While syntax is a crucial aspect of language (Garrett, 1976). how sensitive or in- variant lanugage models are to syntactie informa- tion is complex topic (Pham et al., 2021; Sinha al., 2021; Pupadimitriou et al., 2022; Abdou et al., 2022). In the domain of transfer learning, we investigate a set of syntactic transformations that isolate syntactic word-order shifts from the other factors that differ between languages. We bound our syntactic transformation experiments with a random shuffle control, where no word order in- formation from the original language can be used decode the new language. We also do the sim- ple, but drastic baseline of reversing the order of all of the words in the input. In order to test the effect of more realistic syntactic changes, we trans- form the English data into t-Englishes that follow the word-order statistics of other language. Us- ing the Galactic Dependencies package (Wang and Eisner, 2016) with Stanza (Qi et al., 2020) transform our corpora to match the ordering of words in noun phrases and verb phrases of French ([Nr. vi)) and Japanese ((Nja: Vja)) and also per- form a mixed transformation with French noun order and Japanese verb order ((Ni, Vja)).
How much does tokenizer quality mat- ter, if the price of a better tokenizer is having to reinitialize the whole word embedding matrix? Though quality tokenizers undoubtedly play an important role in multilingual NLP (Rust et al., 2020), we wish to compare the effect of tokenizer quality when the word identity alignment prob- lem remains constant. While re-initializing the embedding matrix, we compare the effects of the original RoBERTa a tokenizer, to two tokenizers that produce low-quality tokenizations for English text: the French FlauBERT (Le et al., 2020) and the Dutch DutchBERT (de Vries et al., 2019). The non- English tokenizers used tokenize Linglish text simulate the effect of having a bad, non-language- specific tokenizer in the low data regime (see Ap- pendix B for statistics on how the different tokeniz- ers work on English).
Tokenizer
Word identity alignment Previous works have consistently found that good embeddings are crucial for enabling effective crosslingual trans- fer (Tran, 2020; Artetxe et al., 2020). However,
Results
'For comparison, the pretraining data for RoBERTa con- tains 3.3B tokens, so 15M tokens is about 0.45% of its pre- training data. This is comparable to the size of the OSCAR corpus for Yiddish (Ortiz Suárez et al., 2019).
We present the main results of our transfer exper- iments. Our experimental details (c.g. hyperpa- rameter choices) with a per-task breakdown of t-

---

Syntactic transformations. Direct fine-tune on GLUE
Syntactic transformations,
Continued pretraining on 1-English
Mean accuracy on I-GLUE
80-
-8,09
77.41
-10.55
Mean accuracy t-GLUE
80-
2.12
.2.29
-3.22
0.00
40
40-
0-
Fr Noun Vr
Ja Noun
Noun
Reverse
Random
Originll
Fr Noun Verb
Ja Noun
Fr Noun
Reverse
Ransom Order
Or Orginil
Ja Verb
J5 Verb
Order
Order
Ja Verb
Ja Vero
Order
Figure Models are largely able to adapt to syntactic shifts with minor drops in performance. Averaged GLUE scores for t-Englishes with syntactic shifts. Realistic syntactic shifts slightly impact downstream performance, while reverse and random order impact performance more significantly. Error bars represent 95% confidence intervals over random seeds. Results are depicted for RoBERTa, are consistent for models that we tested: RoBERTa. DeBERTa, and XLM-R (all results in Figure in Appendix A).
Reinitialize embedding matrix. fine-tune on t-GLUE
Reinitialize embedding matrix, Continued pretraining on t-English
0.00
0.06
Mean ancuracy on I-GLUE
Mean accuracy t-GLUE
-1538
114.88
-15.01
-14
-14.41
80-
60-
-31.90
-30.78
-30.81
-32.39
80-
40-
DUichBERT
Original, ambs
Onginal (Contro))
BERT Tokeniber
ABERT
RlaBERT DutchBERT
Original, renit shuflle
Original,
Original reintt
Tokeeizer Tokenizer
Tokenzer
Tokenizer Tokerizer
emita
embs
embs
(Control)
Figure 3: Token embedding transformations are hard to recover from, regardless of tokenizer. Averaged GLUE scores for t-Englishes with word identity perturbations. Any embedding reinitialization or shuffling, regardless of the tokenizer ultimately used, has drastic effect downstream performance. Error bars represent 95% confidence intervals over 3 random seeds. Results are depicted for RoBERTa, but are consistent for all 3 models that we tested: RoBERTa, DeBERTa, and Xaall results in Figure in Appendix A).
GLUE performance as well as additional results on DeBERTa and XLM-R are included in Appendix A.
with continued pretraining on 15M tokens of the transformed t-English data. These results suggest that syntactic shifts have real but limited impact crosslingual transfer when disentangled from vocabulary learning effects.
4.1 Syntax matters, but training can mostly recover
Word order permutations have an effect on model performance, but the models that we test can recover relatively well from linguistic word or- der permutations when there are no vocabulary confounders. As shown in Figure 2. simply by fine-tuning GLUE RoBERTa can recover from linguistic-style syntactic shifts relatively well, though this significantly worse for random word order permutations that have no consistency or syn- tactic backing. These differences are all lessened
4.2 Good embeddings matter most, bad embeddings can ruin a good tokenizer
Looking at the isolated effect of vocabulary, we find that in the low-data transfer regime the model has hard time reconstructing reinitialized embedding matrix. As shown in Figure 3, reinitializing the embedding matrix causes huge failures for the di- rect fine-tune case, and the quality of the tokenizer (language-bespoke versus not) do not have an effect

---

Fine-tune XLM-R on different
Despite looking at general factors that differ be- tween languages, and using empirical syntactic pat- terns from non-English languages, the fact remains that all of our experiments are centered on English and t-Fnglishes, and this may introduce English- centric biases.
XNLI test sets
80
Mean accuracy on trarsformed XNLI test set
60
40
05
2h
on
Jh
Our scope is mainly restricted English LLMs (vs other languages), three components of crosslin- gual shifts (vs other potential factors), and GLUE tasks (vs other kinds of NLP tasks). While our experiments are not an exhaustive list of linguistic properties that affect cross-lingual transfer, we aim to focus on crucial factors that change between lan- guages, grounded by the literature. Our paradigm is extensible to other model architectures while we fo- cus on RoBERTa in this paper with additional results on DeBERTa and XLM-R included in Appendix A.
Embeddings
Embeddlnggs
Figure 4: Our findings generalize to fine-tuning on non-English datasets. Fine-tuning on three different XNLI datasets yields similar findings the English GLUE findings: models can recover from the most extreme syntactic case (random ordering) much more effectively than from any of the embeddings-related perturhations. This indicates our findings are not related proper- ties specific to the English language.
Ethies Statement
Our experiments provide a controlled environment to test hypotheses about what influences cross- lingual transfer. However, English-based exper- imentations affecting other languages should not be used to determine courses of action for low- resource NLP without supplementary in-language experiments.
beyond this. Our results suggest that tokenization may thus be a "lower-order bit" for crosslingual transfer, which has little impact until good word embeddings are learned. In the direct fine-tuning case, shuffling the word embedding matrix is sig- nificantly better than reinitializing the embeddings, though this difference disappears with continued pretraining.
Acknowledgements
We would like to thank Nelson Liu, Mirac Suzgun, and Tolúlopé Ógúnrêmí for useful discussions and comments on drafts. This research was funded in part by NSF award number IIS-2128145.
5 Conclusions
In this paper, propose paradigm to study crosslingual transfer through transforma- tions which simulate and disentangle the linguistic changes across languages. Our results suggest that solving the embedding alignment problem is the "high-order bit" for crosslingual transfer: has the largest impact on finetuning performance and is the least improved by continued pretraining. Thus, fu- ture progress solving this problem in large-scale transformers may have outsized impact.
References
Mostafa Abdou, Vinit Ravishankar, Artur Kulmizev, and Anders Søgaard. 2022. Word order does matter and shuffled language models know it. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 6907-6919.
Mikel Artetxe. Gorka Labaka, and Eneko Agirre. 2018. Generalizing and improving bilingual word embed- ding mappings with a multi-step framework of linear transformations. In Thirty- second AAAI conference artificiall intelligence.
Limitations
Our paper is about multilinguality in NLP. How- ever, using multiple natural languages would make it impossible disentangle different factors. By us- ing controlled variants of a single language, we can create controllable environment to investigate and understand the factors that affect real cross- lingual transfer in a multilingual setting.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623-4637.

---

Alexandra Chronopoulou, Dario Stojanovski. and Alexander Fraser. 2020. Reusing a pretrained lan- guage model languages with limited corpora for unsupervised nmt. In Proceedings of the 2020 Con- [erence on Empirical Methods in Natural Language Processing (EMNLP), pages 2703-2711.
Hang Le, Loic Vial, Jibril Frej, Vincent Segonne. Max- imin Coavoux, Benjamin Lecouteux, Alexandre Al- lauzen, Benoit Crabbé, Laurent Besacier, and Didier Schwab. 2020. Flaubert: Unsupervised language model pre-training for french. In Proceedings of the 12th Language Resources and Evaluation Confer- ence, pages 2479-2490.
Kevin Clark, Minh-Thang Luong, Quoe V Le. and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555.
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Na- man Goyal, Marjan Ghazvininejad, Luke Zettle- moyer, and Madian Khabsa. 2023. XIm-v: Overeom- ing the vocabulary bottleneck in multilingual masked language models. arXiv preprint arXiv:2301. 10447.
Wietse de Vries, Andreas Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and BERTje: A Dutch BERT
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer. and Veselin Stoyanov. 2019. ROBERTa: robustly optimized BERT pretraining approach. arXiv preprint ArXiv: 1169..
Malvina Nissim. 2019.
Model.
Ameet Deshpande, Partha Talukdar, and Karthik Narasimhan. 2022. When is BERT multilingual? isolating crucial ingredients for cross-lingual transfer. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 3610-3623, Seattle, United States. Association for Computational Linguistics.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod- els. arXiv preprint arXiv: 009407783.
Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When being un- seen from mbeett is just the beginming: Handling new languages with multilingual language models. In NAACL-HILT 2021-2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 181.0:1810.04805.
Philipp Dufter and Hinrich Schütze. 2020. Identifying necessary elements for bert's multilinguality. arXiv preprint 2tth::2000000999
Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. 2021. Small data? no problem! exploring the viability of pretrained multilingual language models for low- resourced languages. In Proceedings of the 1st Work- shop on Multilingual Representation Learning, pages 116-126. Punta Cana, Dominican Republic. Associa- tion for Computational Linguistics.
Merrill F Garrett. 1976. Syntactic processes in sentence production. New approaches to language mecha- nisms, 30:231-256.
Tolulope Ogunremi, Dan Jurafsky. and Christopher Manning. 2023. Mini but mighty: Efficient multi- lingual pretraining with linuustically-iifoomee selection. In Findings of the Association for Compu- tational Linguistics: EACL 2023, pages 1251-1266. Dubrovnik, Croatia. Association for Computational Linguistics.
Evangelia Gogoulou, Ariel Ekgren, Tim Isbister, and Magnus Sahlgren. 2021. Cross-lingual trans- fer of monolingual models. arXiv preprint arXiv:2109.07344
data
Karthikeyan K. Zihan Wang. Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual bert: An empirical study. In International Conference on Learning Representations.
Akintunde Oladipo, Odunayo Ogundepo, Kelechi Ogueji, and Jimmy Lin. 2022. An exploration of vocabulary size and transfer effects in multilingual language models for african languages. In 3rd Work- shop on African Natural Language Processing.
Taku Kudo and John Richardson. 2018 Sentencepiece: simple and language independent subword tok- enizer and detokenizer for neural text processing. arXiv preprini
Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipelines for process- ing huge corpora on medium to low resource infras- tructures. Proceedings of the Workshop on Chal- lenges in the Management of Large Corpora (CMLC- 7) 2019. Cardiff, 22nd July 2019, pages 16, Mannheim. Leibniz- Institut für Deutsche Sprache.
Alexandre Lacoste. Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learn- ing of language representations. arXiv preprint arXiv: 1942.
Isabel Papadimitriou, Richard Futrell, and Kyle Ma- howald. 2022. When classifying arguments, BERT doesn't care about word order...except when it mat- ters. In Proceedings of the Society for Computation

---

Linguistics 2022, pages 203-205, online. Associa- tion for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv: 1508. 07990.
Isabel Papadimitriou and Dan Jurafsky. 2020. Learn- ing music helps you read: Using transfer study linguistic structure in language models. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6829-6839.
Koustuv Sinha, Robin Jia, Dieuwke Hupkes. Joelle Pineau, Adina Williams, and Douwe Kiela. 2021. Masked language modeling and the distributional hy- pothesis: Order word matters pre-training for little. arXiv preprint arXiv: 10406664..
Vaidehi Patil, Partha Talukdar, and Sunita Sarawagi. 2022. Overlap-basee based vocabulary generation im- proves cross-lingual transfer among related lan- guages. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 219-233, Dublin, Ireland. Association for Computational Linguisties.
Alex Tamkin, Trisha Singh, Davide Giovanardi, and Noah Goodman. 2020. Investigating transferability in pretrained language models. Findings of the As- sociation for Computational Linguistics: EMNLP 2020.
Ke Tran. 2020. From english to foreign languages: Transferring pre-trained language models. arXiv preprint 2rxiv:2002..7.36.
Jonas Pfeiffer, Ivan Vulić, Tryna Gurevych, and Se- bastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Tat Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 7654-7673, Online. Association for Computa- tional Linguistics.
Giorgos Vernikos and Andrei Popescu-Belis 2021. Sub- word mapping and anchoring across languages. In Findings of the Association for Computational Lin- guistic:: EMNLP 2021, pages 2633-2647.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLPP. Analyz- ing and Interpreting Neural Networks for NIP, pages 353-355.
Jonas Pfeiffer, Ivan Vulié, Tryna Gurevych, and Sebas- tian Ruder. 2021. UNKs everywhere: Adapting mul- tilingual language models to new scripts. In Proceed- ings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 10186-10203, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Dingquan Wang and Jason Eisner. 2016.
The galac- tic dependencies treebanks: Getting more data by synthesizing new languages. Trunsactions of the As- sociation for Computational Linguistics, 4:491-505.
Thang Pham, Trung Bui, Long Mai, and Anh Nguyen. 2021. Out of order: How important is the sequen- tial order of words in sentence in natural language understanding tasks? In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pages 1145-1166 Online. Association for Computa- tional Linguistics.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, al. 2016. Google's neural machine translation system: Bridging the gap between human and machine trans- lation. arXiv preprint 1xXiv:1609.0000 00194.44444..
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations.
Zhengxuan Wu, Nelson F Liu, and Christopher Potts. 2021. Identifying the limits of cross-domain knowl- edge transfer for pretrained models. arXiv preprint arXiv:2104.0844
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. 2019. Transfusion: Understanding transfer learning for medical imaging. Advances in neural information processing systems, 32.
A Results on other models
We present the results in Figures 2 and 3 for two more models: DeBERTa and the cross-lingual model XLM-R:
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual us- ing knowledge distillation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4522-4525.
B Sequence Length Distribution
As described in Section 3.1, we try four differ- ent tokenizers to substitute for our RoBERTa (Liu et al., 2019) model that uses the Byte-Pair En- coding (BPE) (Sennrich et al., 2015) tokenizer. Specifically, we substitue with the WordPiece to- kenizer (Wu et al., 2016) used by BERT (Devlin
Phillip
Rust, Jonas Pfeiffer. Ivan Vulić. Sebastian Ruder, and Tryna Gurevych. 2020. How good is your tokenizer? the monolingual performance of multilingual language models. arXiv preprint 15513.

---

Syntactic transformations, Direct fine-tune on GLUE
Mean accuracy On H-LLUE
6.00
-7.00
-12.43
3.
-4.25
-10.00
80+
Model
RoBiERla XLM-R
80-
Fr Noun Fr Vert
Ja Noun Ja Vero
Noun Vero
Reverse
Random Order
Original
Order
Figure 5: Models are largely able to adapt to syntactic shifts with minor drops performance. Results for the embedding transformations shown for RoBERTa Figure 2, for all models that we tested: RoBERTa, DeBERTa. and XLM-R.
Reinitialize embedding matrix, Direct fine-tune t-GLUE
30
0.00
Mean acctracy on L-GLUE
0.00
-17.00
Model
-31.17-29.D
-31.00
-5.17
20
311.00
35.99 28./1
-36.e-21.0
DeBERTO
BERT Tokerizer
ABERT Tokenizer
RauBERT Tocenkzer
DunchBERT Tokenizer
Original, resli
Orignal shulll
Original no (Control)
Figure Token embedding transformations hard recover from. Results for the embedding transformations shown for RoBERTa in Figure 3, for all models that we tested: RoBERTa, DeBERTa, and XLM-R.
al., 2018) (i.e., BERT Tokenizer in Table and the SentencePiece tokenizer (Kudo and Richard- son, 2018) used by Albert (Lan et al., 2019) (i.c.,
C Training Set-up Details
Downstream Task. We use the GLUE bench- mark (Wang et al., 2018) evaluate model perfor- mance, which covers nine different NLP tasks. We report scores on the development sets for each task by fine-tuning our pre- trained or mid-tuned models. We fine-tume tie-tune for 5 epochs for the smaller datasets (WNLI and MRPC) and 3 epochs for the others. For the performance metrics, we use Matthew's 's Correlation for CoLA, Pearson correlation for STS- B, and accuracy for all the other datasets.
Albert Tokenizer in Table 1). Additionally, we substitute with two new nng-Esh tokenizers including the French FlauBERT (Le et al., 2020) (FlauBERT Tokenizer in Table 1) and the Dutch DutchBERT (de Vries et al., 2019) (DutchBERT To- kenizer in Table 1). As shown in Figure we plot the distributions of sequence lengths as a measure the heterogeneity introduced by new tokeniz- ers to ensure variences across tokenized sequence lengths. Specifically, we see there are inferior tok- enizers such as F FlauBERT Tokenizer with a 22.15% increase in sequence length. Our results are con- sistent with previous findings (Rust al., 2020) where sequence length distributions closer.
Hyperparameter and Infrastructure. For each of the mid-tuning and fine-tuning experiments, we collect averaged results from runs with distinct random seeds. We tune our models with learn- ing rates {2c 4e 5), and report the best results from these two learning rates. Fine-tuning with GLUE tasks takes about 8 hours on NVIDIA

---

Sequence Lengths
50
Lengths
40
30
RoBERTa
BERT
Albert
FRaBERT
DetchBERT
Figure 7: Distributions of sequence lengths by different tokenizers.
Titan 12G GPUs. Mid-tuning with our subset of WiKM corpus takes about 18 hours with the same infrastructure.
Detailed GLUE Task Performance
Table 2 shows performance break-down for individ- ual GLUE task under different transformations as described in Section 3.1. The individual t-GLUE and GLUE results included in Table 2. We find a consistent picture across most of the tasks, with some interesting effects like CoLA (which is more syntax-sensitive) being impacted more by syntactic shifts.

---

Original
Token Swap
Word Swap
Reini(FEm)
Bert
Albert
FlauBERT
DutchBERT
Ranlom
Revers
(Nu. Ver
(N.. Vaa
(NG. Via!
50000)
300,000
.00,00)
.01,011
.30.00)
CaLAAg
59,011
06.305
010011
22(04)
.35,013
47001)
MNLI
88.00)
34(.0L)
20,08)
S3x,03)
54,01)
53,01)
.57,01)
68.000
82:00)
85,,00
.50.00)
86L000
A5.000
,
733,00
82.00)
86LD00
MRPC
88L01)
.81 (00)
681 08)
68,000
68.00)
6SL00)
.26.01)
.77.013
77(01)
85(,02)
05)
36L01)
S3 000000
MRPC.
$70.00
.33(.00)
S0,04)
.79,011
82(01)
30,011
.84.01)
.78,013
.1.011
.71,013
.51.00
871013
(78,01)
88.00)
QNLI,
93,000)
.33.00)
52101)
82,,00)
83(00)
$2100)
52v.00)
81(.00)
88,00)
911,00)
911
920000
91 100
QOP
911,00)
.77(00)
.77000)
.77,000
.76.00)
75L00)
.85,000
866.00)
90,00)
.910000
.00,00)
911.00)
900,000
QQP,
RTE
65(,02)
511.00)
511,03)
S3v,00)
53(,00)
53(01)
.54.00)
566.2)
57(01)
(20
/5Xx (0)
611.0)
.9K,05)
RTF.,.
.67,011
$6(,0)
53(01)
S4,,11)
-57(01)
59(,02)
.5T(01)
.57
59((22)
.58(02)
JAN
64(,05)
66%
SST-2g3.
.33.01)
X5(01)
856001
XSLD0)
X2.00)
.82.011
.11.013
28000)
.93,00)
.90,00)
93LD0)
.2),000
ST&-B
39.00)
06000 01)
06(00)
(66,02)
(99)
(2()22)
(T4,01)
.77.80)
87(110))
887
88(D0)
880
WNLI
.56.K0
360.00 00)
56000)
.50,000
56.000
50,033 03)
.50,000
.66,013
55,01)
.56,01)
(0)
560
.56,01)
WNLL,
-56.11)
531
53(05)
S3L,03
55(02)
51()7)
.66,00)
56..8)
55(01)
.51,
366
56((0)
.53,05)
Table 2: GLUE scores for t-English with different types of interventions including scrambled word identities, syntactic shifts, and tokenizer substitutions with standard deviation (SD) for all tasks across 3 distinct runs with different random seeds. The scores with original English sentences are included for comparison. c.p. indicates finetuning results with continued pretrained models.


Reviews:
[{"review_id": "JmjGRtYF1l", "reviewer": "Reviewer_4H2z", "summary": "", "questions": "q1. what sort of other factors can NLP community focus on to continue the experimentation?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We *thank* the detailed comments and suggestions. We are encouraged that you find that our paper is interesting! We have addressed your detailed comments below. Here we address your detailed comments, which are helping us to revise the paper and chart out future directions. Additionally, we will incorporate all the new results in the response into the revised version.\n\n> **Q1:** “what sort of other factors can NLP community focus on to continue the experimentation?” \n\n**A1:** This is an excellent question! With our experiments focus on shifts in tokenization, syntactic shifts, and vocabulary shifts, **there are many other linguistic shifts one could investigate using our method**. For example: rewriting a language to have richer or weaker morphology (perhaps delegating the rewriting task to a language model), changing certain kinds of words but not others (e.g. to isolate the impact of articles vs nouns vs verbs), or by changing words that are most likely to change between different dialects.\n\n**In terms of additional analytical experiments, future research could look at a more fine-grained version of our experiments to gain additional insights.** For example, while we have explored extensive shifts between languages where an entirely new vocabulary is required, our t-Language methodology is versatile enough to be utilized in a broader array of contexts. For example, one could consider relearning only a smaller fraction of embeddings, either selected randomly or based on word frequency, to study a scenario approximating more closely related languages.\n\n> **Q2:** “the scope of this paper was limited to English language” \n\n**A2:** This is a great suggestion. Though our experiments are meant to simulate the effect of transferring to different languages in a controlled setting, we still only use English as the source of the t-languages. However, **we do use multilingual source models**: in the current set of experiments, we have replicated our results on English pretrained model RoBERTa (English -> t-English) on XLM-R, which is a multi-lingual model trained with more than 100 languages, as well as DeBERTa (see Appendix A.4). \n\nTo address the question of non-English source languages, **we ran additional experiments with 2 non-English languages** (**Spanish** and **Chinese**). We examine 4 conditions: (1) original (non-transformed); (2) reinitialized embedding matrix; (3) token identity swap (i.e., swapping rows of the embedding matrix); (4) random word order. Our results are shown in the following table:\n\n| XLM-R | Original | Reinit Embeddings | Token Swap | Random Order |\n|---------|---------|---------|---------|---------|\n| XNLI-en (English)  | 0.84  | 0.33  | 0.33  | 0.80  |\n| XNLI-es (Spanish)  | 0.81  | 0.33  | 0.33  | 0.76  |\n| XNLI-zh (Chinese)  | 0.78  | 0.33  | 0.35  | 0.76  |\n\n**Consistent to our current findings, word embeddings play a major role in crosslingual transfer for both Spanish and Chinese.** Additionally, drastic syntactic shifts only cause minimum drops in terms of performance which is what we found with our current experiments. We will include these results in the next revision.\n\n> **Q3:** “Inability to use the same setup for analyzing transfer of LM from one language to another.”\n\n**A3:** **This is definitely right**, in the real world many factors change simultaneously between two different natural languages (e.g., English to Spanish). **However, this is one of the key motivations of our work**: our controlled experiments enable us to isolate the effect of a single factor and obtain a cleaner answer about its impact on knowledge transfer. We hope for our paper to act as an analysis of what is difficult when doing crosslingual transfer, rather than a practical setup for doing crosslingual transfer. \n\nLastly, we will revise our draft to correct all the typos and missing references. We will release our code for training as well as the code to generate t-Englishes (currently included in our supplementary material) upon publication.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Good experiment setup for analysis by focusing on one variation at a time. The results can be extrapolated to actual language transfer helping the NLP community for further research in multilingual models. \n2. Interesting insight of knowing that ”good embeddings matter the most.”", "reasons_to_reject": "1. As the scope of this paper was limited to English language, we can’t be certain the results would scale to other languages. \n2. Inability to use the same setup for analyzing transfer of LM from one language to another. This sort of transfer experiment can further help in scaling our results to other languages.", "questions_to_authors": "q1. what sort of other factors can NLP community focus on to continue the experimentation?", "paper_topic_and_main_contributions": "The paper is about studying the analysis of transformations in cross lingual transfer settings. The contribution this paper makes is to propose a controlled experiment setup to understand the important factors that can have major impact on cross-lingual transfer. The problem: Practitioners are unsure on what factors to emphasize on during cross-lingual transfer of Language Models (LMs). The solution proposed: The authors simulate a cross-lingual transfer by transforming English LM to transformed English (on one axis of variation). These variations are considered as factors that can effect Language Models. By doing one axis of variation at once, the experiments are controlled and help in analyzing the results. These factors are Word-order syntactics, word identity alignments and tokenization variations. The authors discover that word identity alignments has the major impact on a model’s capability of performing well on a different language.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "typos: \n1. Line 17: had → hard \n2. Line 43: missing “to” \n3. Line 170: missing reference \n4. Line 194: may → may be \n5. Line 197: pretraiend → pretrained \n6. Line 250: missing comma \n7. Line 255: dot → do"}}, {"review_id": "YYgOvcDgeX", "reviewer": "Reviewer_h6Fy", "summary": "", "questions": "A: The work tries to assign importance to language characteristics by studying the performance of models on transformations of language. However, these \"characteristics\" are not impacted in the same way by Fine-tuning or even Pre-training. Can the authors explain how this generalization follows given these differences?\nB: As a practical follow up to (A), 0-layer embeddings are notoriously hard to influence. To treat them on par with word-order for example, the rest of the network could be frozen (i.e, stop gradients). Would this fix the losses seen when re-initing embeddings?\nC: How are sentinel tokens handled such as [CLS] when shuffling the emb-matrix? If naively shuffled it will lead to significant errors in classification / QA / GLUE tasks.\nD: In random ordering of words-in-sequence, is care taken to always re-order sentences in the same way? I.e, a shuffling map is generated once and applied to all inputs? Or is input shuffling completely independent from sentence to sentence?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We *thank* the reviewer for the detailed review! We are encouraged that you find that our method is a simple and effective way of studying crosslingual transfer. We have addressed your detailed comments below. Additionally, we will incorporate the new results in our response into the revised version.\n\n> **Q1:** Code is not public (or at least I missed the reference)\n\n**A1:** Our code can be found in the `supplementary zip file. **We will release our code for training as well as to generate t-Englishes upon publication.** Our code is largely based off HuggingFace’s training code to ensure good reproducibility of any experiment.\n\n> **Q2:** Can the authors explain how this generalization follows given these differences?\n\n**A2:** We want to clarify that t-English aims to isolate one transformation at a time (e.g., Syntactic Shifts), and study its impact on crosslingual transfer. We applied the transformation to all downstream tasks in the same way. **We agree that the impact on each task could differ, and we provided per-task breakdown in Table 2 in the Appendix.** Overall, the trend is consistent across tasks.\n\nBesides the effect on **Finetuning**, we also have parallel experiments with **Pretraining-then-Finetuning** (See bottom panel of Figure 2) where we hope the model adapts to t-Englishes with continued pretraining (i.e., standard masked language modeling). While **Pretraining-then-Finetuning** improves model performance, it does not close the gap completely, as shown through all of our experiments.\n\n> **Q3:** 0-layer embeddings are notoriously hard to influence. To treat them on par with word-order for example, the rest of the network could be frozen (i.e, stop gradients). Would this fix the losses seen when re-initing embeddings?\n\n**A3:**  This is an interesting suggestion! **We ran additional experiments by reinitializing the embedding matrix and freezing the rest of the network while training the embeddings across 9 tasks and 3 different models** (RoBERTa, DeBERTa, XLM-R). Our results are shown in the following table:\n\n| Model | Original | Reinit + Train All Layers | Reinit + Train Only the Embedding Layer |\n|---------|---------|---------|---------|\n| RoBERTa  | 0.80  | 0.48  | 0.40  |\n| DeBERTa  | 0.82  | 0.45  | 0.46  |\n| XLM-R  | 0.72  | 0.43  | 0.37  |\n\n**Our results show that tuning all layers is more effective.** Note that this is directly finetuning. In addition, we also tried this with continued pretraining on RoBERTa, and found it did not help in terms of downstream task performance. We will include these additional results in the next revision, thanks for the insightful suggestion!\n\n> **Q4:** C: How are sentinel tokens handled such as `[CLS]` when shuffling the emb-matrix?\n\n**A4:**  **We ran additional experiments where we only reinitialized the `[CLS]` token embeddings for three different models**: RoBERTa, DeBERTa, and XLM-R. We then evaluated these modifications across nine tasks. The results are tabulated below:\n\n| Model | Original | Reinit All Tokens + Train All Layers | Reinit Only `[CLS]` Tokens + Train All Layers |\n|---------|---------|---------|---------|\n| RoBERTa  | 0.80  | 0.48  | 0.78  |\n| DeBERTa  | 0.82  | 0.45  | 0.82  |\n| XLM-R  | 0.72  | 0.43  | 0.70  |\n\nInterestingly, when only reinitializing the `[CLS]` token, DeBERTa does not have any performance drops compared to other two models. Furthermore, the significant performance dips observed when reinitializing all embeddings imply that the `[CLS]` token embedding is not the only determining factor in this context.\n\n> **Q5:**  In random ordering of words-in-sequence, is care taken to always re-order sentences in the same way?\n\n**A5:**  The **random ordering experiments are designed to be the most challenging setting** where no information about ordering can be used by the network; thus, we apply different random permutations each time (i.e., training and testing orderings are different). This enables us to baseline how important non-structural information is to the model, so we can contextualize our other experiments, such as reversed ordering and reordering based on Galactic Dependencies (Table 1) which apply transformations systematically\n\nLastly, we will revise our draft to correct the typos and missing references mentioned by the reviewer, thank you very much for all of your suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea is simple. Simple ideas are easier to understand and build on.\n2. The work aggregates (as it should!) over multiple models to determine importance of language characteristics.\n3. This is an empirical study that emulates a reasonably sized problem (even if with smaller models than SoTA).", "reasons_to_reject": "1. Code is not public (or at least I missed the reference).  Having public code allows the research community to verify and build on these results. (EDIT: No longer true after rebuttal, as the authors have promised to release it).\n2. There are several nuanced factors that are not taken into consideration (see A, B) in questions for authors.", "questions_to_authors": "A: The work tries to assign importance to language characteristics by studying the performance of models on transformations of language. However, these \"characteristics\" are not impacted in the same way by Fine-tuning or even Pre-training. Can the authors explain how this generalization follows given these differences?\nB: As a practical follow up to (A), 0-layer embeddings are notoriously hard to influence. To treat them on par with word-order for example, the rest of the network could be frozen (i.e, stop gradients). Would this fix the losses seen when re-initing embeddings?\nC: How are sentinel tokens handled such as [CLS] when shuffling the emb-matrix? If naively shuffled it will lead to significant errors in classification / QA / GLUE tasks.\nD: In random ordering of words-in-sequence, is care taken to always re-order sentences in the same way? I.e, a shuffling map is generated once and applied to all inputs? Or is input shuffling completely independent from sentence to sentence?", "paper_topic_and_main_contributions": "This paper experiments with systematic transformations to language in datasets to determine the importance of various language characteristics in Finetuning. This idea is particularly useful to low resource languages.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 161: Million spelt wrongly\nLine 170: incorrect citation ?", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}, {"review_id": "cJ78t52rG0", "reviewer": "Reviewer_qFy2", "summary": "", "questions": "- Question A: in Section 1, what are the rationale behind random word-order and reversed word-order?\n- Question B: in Section 1, what are the rationale behind shuffling the rows of the layer 0 word embedding matrix?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We *thank* the reviewer for the detailed comments and suggestions. We are glad that you find our paper interesting! Here we address your detailed comments, which have helped us further clarify the motivations behind our experiments and chart out future directions. Additionally, we will incorporate all the new results in the response into the revised version.\n\n> **Q1:**  in Section 1, what are the rationale behind random word-order and reversed word-order?\n\n**A1:**  In our word order experiments, the Japanese and French Galactic Dependency permutations attempt to capture one kind of syntactic change that would occur in a realistic transfer setting. **The random word order, on the other hand, is a lower-bound baseline which lets us see how much information is even present at all from word order.** The random word order baseline serves to put the Japanese and French numbers in context, rather than be applicable on its own. We see that recovering from a realistic syntactic transformation closes more than half the gap to this lower bound (recovering from random), which is the most difficult transformation. \n\nThe random baseline is non-linguistic, but it is also not a consistent, learnable transformation. **To disentangle these two factors, we add the reverse baseline, which is still non-linguistic as you mentioned, but it is a consistent, learnable transformation.** We see that both random and reverse are worse than the naturalistic transformations, showing that the practically-relevant syntactic transformations are easier to transfer between than the worst baseline cases. Our baselines also show the surprising fact that the worst-case word-order transformation is still easier to recover from than _any_ of the vocabulary/tokenizer transformations. \n\n\n> **Q2:** in Section 1, what are the rationale behind shuffling the rows of the layer 0 word embedding matrix?\n\n**A2:** This is a great question! We tried two methods of ablating vocabulary information: reinitializing the embedding matrix and shuffling the rows (vocabulary vectors) of the embedding matrix. **We did both because we wanted to be sure that we were assessing the importance of vocabulary in the model, and not some artifact of our specific methodology.** \n\nWhile reinitializing the embedding is a good first step for assessing the importance of vocabulary, it puts the model in a space where all input vectors lie very close to zero and are not in the subspace that the model expects them to be in. **Compared to randomly initializing the embedding matrix, shuffling the rows lets us see whether the model is able to “reindex” the rows while the embeddings are wrong but still lie within a plausible “language subspace” for the model.** This is a more realistic test of what would happen in a crosslingual transfer scenario, when we take a model and throw out the tokenizer and fine-tune with a new tokenizer. In such a case, each row in the embedding matrix would be in a reasonable subspace and correspond to some word embedding for the old tokenizer, but would be assigned to a different word in the new tokenizer. \n\nIn addition to the row-based permutation, **we also tried to add a rotation on top of the original embedding matrix, and we also found RoBERTa struggles to recover the original embeddings.** Taking these results together, LLMs are very sensitive to the “language space” laid out by the embedding matrix.\n\n> **Q3:** it was not verified with tokenizers of languages far from English such as Japanese.\n\n**A3:** This is an excellent question! **We ran additional experiments by using Chinese tokenizer from `bert-base-chinese` and evaluate crosslingual transfer across 9 tasks and with 3 different models** (RoBERTa, DeBERTa, XLM-R). Our results are shown in the following table:\n\n| Model | Original English Tokenizer | French Tokenizer | Chinese Tokenizer |\n|---------|---------|---------|---------|\n| RoBERTa  | 0.80  | 0.49  | 0.46  |\n| DeBERTa  | 0.82  | 0.46  | 0.46  |\n| XLM-R  | 0.72  | 0.43  | 0.43  |\n\n**Our results suggest that the performance drops are similar between the French tokenizer and Chinese tokenizer.** These results strengthen our conclusion that a good initialization of embeddings matters the most, rather than tokenizer quality. Note that Chinese tokenizer did result in much longer sequence length compared to other tokenizers since English words are mostly broken down to individual characters. We will discuss this set of results in detail in our next revision.\n\n> **Q4:**  Experimental results may only be valid for the dataset used.\n\n**A4:** **GLUE composes 9 tasks together**, covering a variety of domains, including sentiment analysis, NLI, and QA. This provides us with a simple yet rich enough testbed to verify our claims about crosslingual transfer. However, we agree that it would be interesting to validate this paradigm with other benchmarks, especially those non-English ones. We have started this process and reproduced some of our results. \n\nSpecifically, **we ran additional experiments with 2 non-English languages** (**Spanish** and **Chinese**). We examine 4 conditions: (1) original (non-transformed); (2) reinitialized embedding matrix; (3) token identity swap (i.e., swapping rows of the embedding matrix); (4) random word order. Our results are shown in the following table:\n\n| XLM-R | Original | Reinit Embeddings | Token Swap | Random Order |\n|---------|---------|---------|---------|---------|\n| XNLI-en (English)  | 0.84  | 0.33  | 0.33  | 0.80  |\n| XNLI-es (Spanish)  | 0.81  | 0.33  | 0.33  | 0.76  |\n| XNLI-zh (Chinese)  | 0.78  | 0.33  | 0.35  | 0.76  |\n\n**Consistent with our current findings, word embeddings play a major role in crosslingual transfer for both Spanish and Chinese.** Additionally, drastic syntactic shifts only cause minimum drops in terms of performance which is what we found with our current experiments. We will provide new results on this in the next revision.\n \nLastly, we will revise our draft to correct all the typos and missing references. Thanks a lot for your suggestions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposed three factors to be considered for transferring pre-trained language models, and verified them independently. To verify the word-order factor, the synthetic dataset was created. Creating synthetic dataset by systematically manipulating the word-order is a contribution in that it requires much less effort than constructing a dataset in a new language from scratch. To verify the word embeddings factor, a partial shuffling method was proposed in addition to a complete initialization. To verify the importance of tokenizers, different types of tokenizers that were not used in training were utilized for experiments. According to experimental results, it reveals that the cross-language differences in word-order can be resolved with further pre-training or direct fine-tuning on the target task.", "reasons_to_reject": "This paper analyzed three factors that should be considered for the transferring pre-trained language models. The first factor, word-order, was verified with a synthetic dataset. Among the experimental settings, the word-order changes considering the characteristics of other languages is meaningful, but the reversed word-order and randomly shuffled word-order do not reflect any of cross-language characteristics and thus they will not be applicable in practice. In addition, it would be beneficial to give some real examples (not synthesized ones) where the word-order factor was applied so that readers can refer to their own cases. The second factor, word embeddings, was verified with re-initializing them and shuffling their rows. The authors need to provide sufficient justification for this. For example, what is the rationale behind shuffling rows, why not shuffle columns, etc. In addition, the shuffling rates need to be investigated to account for different level of language similarities. The third factor, tokenizer quality, was verified with various tokenizers. For cross-language transfer, French and Dutch tokenizers were utilized, but it was not verified with tokenizers of languages far from English such as Japanese.\n\nExperimental results may only be valid for the dataset used. To show generality, authors should include experimental results for several different data sets.", "questions_to_authors": "- Question A: in Section 1, what are the rationale behind random word-order and reversed word-order?\n- Question B: in Section 1, what are the rationale behind shuffling the rows of the layer 0 word embedding matrix?", "paper_topic_and_main_contributions": "This paper analyzed three factors for transferring pre-trained language models to other languages, which are word-order syntactic differences, word identity alignments, and tokenizer quality. To verify their effectiveness, the performance were measured using a transformed GLUE benchmark. To verify the word-order syntactic differences factor, the word-orders of original text were randomly shuffled, reversed, or set according to the characteristics of other languages. To verify the word identity alignment factor, the word embedding matrices were manipulated. To verify the tokenizer quality factor, the tokenizers that was not used for training were utilized. The experimental results showed that manipulating word-orders caused some performance degradation, but it can be partially recovered by fine-tuning or further pre-training. However, manipulating the word embedding matrix and replacing the tokenizer showed a large performance degradations. Thus, this paper showed that the use of well-constructed word embeddings and tokenizer selection are important factors for transferring the pre-trained language models.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- In Section 3.1, there is one missing reference: the sensitivity or invariance studies have shown surprising syntactic invariance in language models is a complex topic (Pham et al., 2021; Sinha et al., 2021; Papadimitriou et al., 2022; ?)"}}, {"review_id": "Q3pPBjCWld", "reviewer": "Reviewer_rSVE", "summary": "", "questions": "A. See reasons to reject\n\nB. How was the word embedding reinitialised? Randomly? Perhaps make it clear in the text.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We *thank* the detailed comments and suggestions. We are glad that you find our paper interesting! Here we address your detailed comments.\n\n> **Q1:** It’s unclear why only French and Japanese word orders are used to transform the English data\n\n**A1:** **The main reason is that we wanted to try a language with radically different word order to English** (Japanese is verb-final and this affects the ordering of many dependencies), as well as French which is broadly more similar to English but has some differences especially in the ordering of the modifiers of nouns. We will clarify this in our next revision. \n\n> **Q2:**  How was the word embedding reinitialized?\n\n**A2:** For each different model that we ran experiments with, **the embeddings are randomly initialized** using the provided initialization function defined with the model architecture. We will clarify this in our next revision.\n\nLastly, we will revise our draft to correct all the typos and missing references.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Clear hypothesis with sound experiments to test it", "reasons_to_reject": "It’s unclear why only French and Japanese word orders are used to transform the English data, but this is somewhat minor", "questions_to_authors": "A. See reasons to reject\n\nB. How was the word embedding reinitialised? Randomly? Perhaps make it clear in the text.", "paper_topic_and_main_contributions": "This paper studies how different aspects of cross-lingual transfer affect transfer performance by transforming English along one aspect and investigate the results.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. L043: compare → comparing\n2. L057: syntactic typologicaly → syntactic and typological\n3. L161: million\n4. L170: missing citation"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper analyzes important factors in crosslingual transfer of pre-trained LLMs. The paper studies three axes of variation using controlled studies. The reviewers agree about the neat experimental set up. Reviewers raised concerns about only English language experiments but this is addressed in the rebuttal.
Decision: Accept-Main