SlowGuess/NAACL_2021_OCR huggingface dataset:
Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. 2018. Adversarial deep averaging networks for cross-lingual sentiment classification. Transactions of the Association for Computational Linguistics, 6:557-570.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b. Emerging cross-lingual structure in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6022–6034, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Dileep A Divekar. 1984. Dc statistical circuit analysis for bipolar ic's using parameter correlations—an experimental example. IEEE transactions on computer-aided design of integrated circuits and systems, 3(1):101-103.
Patricia A Duff. 2015. Transnationalism, multilingualism, and identity. Annual Review of Applied Linguistics, 35:57.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31-36, Melbourne, Australia. Association for Computational Linguistics.
Guillaume Fürst and François Grin. 2018. Multilingualism and creativity: a multivariate approach. Journal of Multilingual and Multicultural Development, 39(4):341-355.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavi-olette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1):2096-2030.
Siddhant Garg and Goutham Ramakrishnan. 2020. BAE: BERT-based adversarial examples for text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Online. Association for Computational Linguistics.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, San Diego, California.
Mira Goral, Monica Norvik, and Bård Uri Jensen. 2019. Variation in language mixing in multilingual aphasia. Clinical linguistics & phonetics, 33(10-11):915-929.
Wenjuan Han, Liwen Zhang, Yong Jiang, and Kewei Tu. 2020. Adversarial attack and defense of structured prediction models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Online. Association for Computational Linguistics.
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. Unicoder: A universal language encoder by pretraining with multiple cross-lingual tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2485-2494.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is BERT really robust? A strong baseline for natural language attack on text classification and entailment. In *The Thirty-Fourth AAAI Conference on Artificial Intelligence*, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8018-8025. AAAI Press.
Aravind K. Joshi. 1982. Processing of sentences with intra-sentential code-switching. In *Coling* 1982: Proceedings of the Ninth International Conference on Computational Linguistics.
Shafiq Joty, Preslav Nakov, Lluis Márquez, and Israa Jaradat. 2017. Cross-language learning with adversarial neural networks. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 226-237, Vancouver, Canada. Association for Computational Linguistics.
Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual bert: An empirical study. In International Conference on Learning Representations.
Braj B Kachru. 1978. Toward structuring code-mixing: An indian perspective. International Journal of the Sociology of Language, 1978(16):27-46.
Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury. 2020. GLUECoS: An evaluation benchmark for code-switched NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3575-3585, Online. Association for Computational Linguistics.
Anatoly V Kharkhurin and Li Wei. 2015. The role of code-switching in bilingual creativity. International Journal of Bilingual Education and Bilingualism, 18(2):153-169.
Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In Proceedings of COLING 2012, pages 1459-1474, Mumbai, India. The COLING 2012 Organizing Committee.
Philipp Koehn. 2010. Statistical Machine Translation, 1st edition. Cambridge University Press, New York, NY, USA.
Guillaume Lample, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In International Conference on Learning Representations.
Anne Lauscher, Vinit Ravishankar, Ivan Vulic, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with multilingual transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Online. Association for Computational Linguistics.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086-6096, Florence, Italy. Association for Computational Linguistics.
Mark Leikin. 2013. The effect of bilingualism on creativity: Developmental and educational perspectives. International Journal of Bilingualism, 17(4):431-447.
Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. 2019. Textbugger: Generating adversarial text against real-world applications. In 26th Annual Network and Distributed System Security Symposium.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei
Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020. Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation. arXiv preprint arXiv:2004.01401.
George C Linderman, Manas Rachh, Jeremy G Hoskins, Stefan Steinerberger, and Yuval Kluger. 2019. Fast interpolation-based t-SNE for improved visualization of single-cell RNA-seq data. Nature methods, 16(3):243-245.
Yaron Matras and Jeanette Sakel. 2007. Grammatical borrowing in cross-linguistic perspective, volume 38. Walter de Gruyter.
Paul Michel, Xian Li, Graham Neubig, and Juan Pino. 2019. On evaluation of adversarial perturbations for sequence-to-sequence models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3103-3114, Minneapolis, Minnesota. Association for Computational Linguistics.
George A. Miller. 1995. Wordnet: A lexical database for english. Communications of the ACM, 38:39-41.
Benjamin Muller, Benoit Sagot, and Djamé Seddah. 2020. Can multilingual language models transfer to an unseen dialect? a case study on north african arabizi. arXiv preprint arXiv:2005.00318.
Carol Myers-Scotton. 1997. Duelling languages: Grammatical structure in codeswitching. Oxford University Press.
Parth Patwa, Gustavo Aguilar, Sudipta Kar, Suraj Pandey, Srinivas PYKL, Bjorn Gambäck, Tanmoy Chakraborty, Thamar Solorio, and Amitava Das. 2020. Semeval-2020 task 9: Overview of sentiment analysis of code-mixed tweets. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain. Association for Computational Linguistics.
Wannaphong Phatthiyaphaibun, Korakot Chaova-vanich, Arthit Suriyawongkul Charin Polpanumas, Lalita Lowphansirikul, and Pattarawat Chormai. 2016. PyThaiNLP: Thai Natural Language Processing in Python.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001.
Pavlin G. Policar, Martin Stražar, and Blaž Zupan. 2019. openTSNE: a modular Python library for t-SNE dimensionality reduction and embedding. bioRxiv.
M. Popel, M. Tomkova, J. Tomek, Łukasz Kaiser, Jakob Uszkoreit, Ondrej Bojar, and Z. Žabokrtsky. 2020. Transforming machine translation: a deep learning system reaches news translation quality comparable to human professionals. Nature Communications, 11.
Shana Poplack. 1980. Sometimes i'll start a sentence in spanish y termino en español: toward a typology of code-switching. Linguistics, 18(7-8):581-618.
Shana Poplack, David Sankoff, and Christopher Miller. 1988. The social correlates and linguistic processes of lexical borrowing and assimilation. Linguistics, 26(1):47-104.
Adithya Pratapa, Gayatri Bhat, Monjit Choudhury, Sunayana Sitaram, Sandipan Dandapat, and Kalika Bali. 2018. Language modeling for code-mixing: The role of linguistic theory based synthetic data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1543-1553.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Byron Reeves, Nilam Ram, Thomas N Robinson, James J Cummings, C Lee Giles, Jennifer Pan, Agnese Chiatti, MJ Cho, Katie Roehrick, Xiao Yang, et al. 2019. Screenomics: A framework to capture and analyze personal life experiences and the ways that technology shapes them. Human-Computer Interaction, pages 1-52.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1085-1097.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856-865, Melbourne, Australia. Association for Computational Linguistics.
Brian Roark, Lawrence Wolf-Sonkin, Christo Kirov, Sabrina J. Mielke, Cibu Johny, Isin Demirşahin, and Keith Hall. 2020. Processing South Asian languages written in the Latin script: the Dakshina dataset. In Proceedings of The 12th Language Resources and Evaluation Conference (LREC), pages 2413-2423.
Uma Roy, Noah Constant, Rami Al-Rfou, Aditya Barua, Aaron Phillips, and Yinfei Yang. 2020. LAReQA: Language-agnostic answer retrieval from a multilingual pool. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language
Processing (EMNLP), pages 5919-5930, Online. Association for Computational Linguistics.
Masoud Jalili Sabet, Philipp Dufter, and Hinrich Schütze. 2020. Simalign: High quality word alignments without parallel training data using static and contextualized embeddings. arXiv preprint arXiv:2004.08728.
Sunayana Sitaram, Khyathi Raghavi Chandu, Sai Krishna Rallabandi, and Alan W Black. 2019. A survey of code-switched speech and language processing. arXiv preprint arXiv:1904.00784.
Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, and Pascale Fung. 2014. Overview for the first shared task on language identification in code-switched data. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 62-72, Doha, Qatar. Association for Computational Linguistics.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. MASS: Masked sequence to sequence pre-training for language generation. In International Conference on Machine Learning, pages 5926-5936.
Shikaripur N Sridhar and Kamal K Sridhar. 1980. The syntax and psycholinguistics of bilingual code mixing. Canadian Journal of Psychology/Revue canadienne de psychologie, 34(4):407.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, Banff, AB, Canada.
Peck Tung Tan. 1988. A description of patterns of code-mixing and code-switching in a multilingual household. In Joseph Foley, editor, New Englishes: The Case of Singapore. NUS Press.
Samson Tan, Shafiq Joty, Min-Yen Kan, and Richard Socher. 2020a. It's morphin' time! Combating linguistic discrimination with inflectional perturbations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2920-2935, Online. Association for Computational Linguistics.
Samson Tan, Shafiq Joty, Lav Varshney, and Min-Yen Kan. 2020b. Mind your inflections! Improving NLP for non-standard Englishes with Base-Inflection Encoding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Online. Association for Computational Linguistics.
Mary W. J. Tay. 1989. Code switching and code mixing as a communicative strategy in multilingual discourse. World Englishes, 8(3):407-417.
Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT — Building open translation services for the World. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (EAMT), Lisbon, Portugal.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 2019. Robustness may be at odds with accuracy. In International Conference on Learning Representations.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2153-2162, Hong Kong, China. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Genta Indra Winata, Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2019. Code-switched language models using neural based synthetic data from parallel sentences. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 271-280, Hong Kong, China. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of bert. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833-844.
Kofi Yakpo. 2015. Code-switching and social change: Convergent language mixing in a multilingual society. Code-switching Between Structural and Sociolinguistic Perspectives, 43:259.
Zhen Yang, Bojie Hu, Ambyera Han, Shen Huang, and Qi Ju. 2020. Code-switching pre-training for neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Online. Association for Computational Linguistics.
Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020. Word-level textual adversarial attacking as combinatorial optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6066-6080.
Huangzhao Zhang, Hao Zhou, Ning Miao, and Lei Li. 2019. Generating fluent adversarial examples for natural languages. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5564-5569, Florence, Italy. Association for Computational Linguistics.
Wei Emma Zhang, Quan Z. Sheng, Ahoud Alhazmi, and Chenliang Li. 2020. Adversarial attacks on deep-learning models in natural language processing: A survey. ACM Trans. Intell. Syst. Technol., 11(3).
A Examples of Real Code-Mixed Text
English + Spanish (Sridhar and Sridhar, 1980)
- The man que vino ayer (who came yesterday) wants ito buyun carro nuevo (a new car).
- El (The) old man esta enojide (is mad).
- Me lve chile ya roasted y peeled ... para hacerlo ells. (I picked up the chile already roasted and peeled for making it there.)
Hindi + English (Bali et al., 2014)
- Befitting reply to mere papa ne maaraa (My father gave a befitting reply)
- ... and the party workers [will] come with me without virodh (protest/objection)
Sarnami + Sranan + Dutch (Yakpo, 2015)
- dus gewoon calat jaiye, tab ego kerki links ki rechts. (So just keep on walking, then [there's] achurch, left or right.)
- kaun wálá damru, ego haigá jaun men ná verfi bhail, ma ego wel hai. (Which [kind of] damru drum, there's one which is not coloured inside but one actually is.)
B Attack Implementation Details
B.1 POLYGLOSS
To reduce the practical running time of our attack, we make use of cross-lingual dictionaries released by Lample et al. (2018) for generating candidate perturbations instead of translating the words in an online fashion. We also use the gold translations of the clean examples when they are available (such as in XNLI), and use the models released by Tiedemann and Thottingal (2020) in the transformers library (Wolf et al., 2019) to translate the examples to other languages. We also cache them in hashtables for fast retrieval.
B.2 BUMBLEBEE
We use the gold translations where available and Tiedemann and Thottingal (2020)'s translation models for the other languages, and align sentences with a neural word aligner (Sabet et al., 2020) backed by XLM-R<sub>base</sub> in our implementation of BUMBLEBEE. Although Sabet et al. (2020) found the "Itermax" algorithm to yield the best performance for their experimental settings, we suggest using the high recall ("Match") algorithm for candidate generation. We inspected the output of both algorithms and found that while Itermax generates more candidates, it also tends to generate noisier alignments compared to Match, which we found to be more conservative.
C POLYGLOSS and CAT Samples
Almost random samples for POLYGLOSS and CAT (we tried to include the sentences with Thai and Hindi characters but did not manage to get them to render with pdflatex).
Table 7: POLYGLOSS adversaries for XLM-  $\mathbf{R}_{\mathrm{base}}$  on XNLI-13.
Table 8: Code-mixed adversarial training examples.
Standard  $\mathrm{XNLI}^3$  comprises 7,500 parallel examples (2,490 dev., 5,010 test) in fifteen languages: English (en), Spanish (es), German (de), Greek (el), Russian (ru), Turkish (tr), Arabic (ar), Vietnamese (vi), Thai (th), Chinese (zh), Hindi (hi), French (fr), Bulgarian (bg), Swahili (sw), and Urdu (ur). The labels are uniformly distributed between the three classes (contradiction, neutral, entailment).
The machine-translated training set<sup>4</sup> of standard XNLI comprises the MNLI training examples in addition to their translations in the same fourteen non-English languages as the dev. and test sets.
XNLI-13 comprises all standard XNLI languages except Swahili and Urdu due to the lack of suitable dictionaries for POLYGLOSS. XNLI-31 comprises all languages in XNLI-13, in addition to another eighteen: Afrikaans (af), Albanian (sq), Catalan (ca), Czech (cs), Danish (da), Dutch (nl), Estonian (et), Filipino (tl), Finnish (fi), Hebrew (he), Hungarian (hu), Indonesian (id), Italian (it), Macedonian (mk), Romanian (ro), Slovak (sk), Swedish (sv), and Ukrainian (uk).
MNLI comprises 392,702 examples in English with the following label distribution: 130,899 entailment, 130,900 neutral, 130,903 contradiction.
$\mathrm{XQuAD}^6$  comprises 1,190 question-answer pairs with guaranteed answers in eleven languages: English (en), Spanish (es), German (de), Greek (el), Russian (ru), Turkish (tr), Arabic (ar), Vietnamese (vi), Thai (th), Chinese (zh), and Hindi (hi). XQuAD examples are drawn from the SQuAD 1.1 development set.
SQuAD 1.1<sup>7</sup> comprises 87,599 question-answer pairs with guaranteed answers in English.
D.2 Metrics
The metric used for XNLI is simple accuracy:
$$
\text {A c c u r a c y} = \frac {\# \text {t r u e p o s i t i v e}}{\# \text {t o t a l}} \tag {2}
$$
The metrics used for SQuAD are  $\mathrm{F}_1$ :
$$
F _ {1} = \frac {2 \cdot \text {p r e c i s i o n} \cdot \text {r e c a l l}}{\text {p r e c i s i o n} + \text {r e c a l l}} \tag {3}
$$
and exact match:
$$
\text {E x a c t M a t c h} = \frac {\# \hat {y} = y}{\# \text {t o t a l}} \tag {4}
$$
D.3 XQuAD Preprocessing
We use Phatthiyaphaibun et al. (2016) to tokenize Thai text, jieba<sup>8</sup> for Chinese text, and split all other languages on whitespace.
Model	Params.	Lr	Bsz	Epochs
XLM-Rlarge	550M	1e-06	64	10
XLM-Rbase	270M	5e-06	64	10
mBERTbase	172M	5e-05	64	2
Unicoderbase	270M	5e-06	64	10
Table 9: Hyperparameters for model fine-tuning on XNLI (MNLI). Number of parameters reproduced from Conneau et al. (2020a).
Model	Learning rate	Batch Size	Epochs
XLM-Rlarge	1e-05	32	3
XLM-Rbase	3e-05	64	2
Table 10: Hyperparameters for model fine-tuning on XQuaD (SQuAD 1.1).
Hyperparameters. Table 9 contains the hyperparameters we used to fine-tune our models on MNLI. We used the hyperparameters suggested by Devlin et al. (2019)<sup>9</sup> for mBERT, the hyperparameters suggested by Bari et al. (2020) for the XLM-R models, and the hyperparameters from Liang et al. (2020) for Unicoder.