SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 3: (1) Evaluation: We report  $\Delta_{(\mathbf{BZ} - \mathbf{BS})}$  and  $\Delta_{(\mathbf{MZ} - \mathbf{BS})}$  ( $\S$  3.3 and Table 2) for transformations on different tasks, averaged over four languages (EN, FR, HI, AR). We report the breakdown for different languages in Appendix B.  $BZ$ , the bilingual zero-shot performance, is reported for reference. (2) Interpreting scores: Smaller (more negative)  $\Delta_{(\mathrm{BZ - BS})}$  implies worse bilingual zero-shot transfer, whereas  $\Delta_{(\mathrm{BZ - BS})} \approx 0$  implies strong transfer.  $\Delta_{(\mathrm{BZ - BS})} \gg \Delta_{(\mathrm{MZ - BS})}$  implies that bilingual pre-training is extremely useful. Scores are highlighted based on their value (lower scores have a higher intensity of red). Cases with strong zero-shot transfer ( $\Delta_{(\mathrm{BZ - BS})} \approx 0$ ) are marked with an asterisk. (3) Trends:  $\mathcal{T}_{\text{trans}}$  exhibits strong transfer on all tasks and languages (high  $\Delta_{(\mathrm{BZ - BS})}$  scores), and bilingual pre-training is extremely useful ( $\Delta_{(\mathrm{BZ - BS})} \gg \Delta_{(\mathrm{MZ - BS})}$ ), implying that zero-shot transfer is possible between languages with different scripts but the same word order.  $\mathcal{T}_{\text{inv}}$  and  $\mathcal{T}_{\text{perm}}$  suffer on all tasks (small  $\Delta_{(\mathrm{BZ - BS})}$  scores) whereas  $\mathcal{T}_{\text{syn}}$  suffers significantly lesser, which provides evidence that local changes to the word order made by Syntax ( $\mathcal{T}_{\text{syn}}$ ) hurts zero-shot transfer significantly lesser than global changes made by Inversion ( $\mathcal{T}_{\text{inv}}$ ) and Permutation ( $\mathcal{T}_{\text{perm}}$ ).
Dataset	Task	Metric
XNLI (Conneau et al., 2018)	NLI	Accuracy
Wikiann (Pan et al., 2017)	NER	F1
UD v2.5 (Nivre et al., 2018)	POS	F1
XQuAD (Artetxe et al., 2020b)	QA	F1
Table 4: XTREME benchmark datasets used for zero-shot transfer evaluation. NLI=Natural Language Inference, NER=Named-entity recognition, POS=Part-of-speech tagging, QA=Question-Answering.
$\beta$  ) differs from its second constituent  $(\beta)$  in that the former produces a derived language which has  $0\%$  sub-word overlap with the original language whereas the latter has a  $100\%$  sub-word overlap.
Results Our results (Figure 2, breakdown in Appendix C) show that zero-shot performance is significantly hurt for composed transformations when compared to its constituents.  $|\Delta_{(\mathrm{BZ - BS})}|$  is much larger for  $\mathcal{T}_{\mathrm{trans}}\circ \mathcal{T}_{\mathrm{inv}}$  when compared to  $\mathcal{T}_{\mathrm{trans}}$  or  $\mathcal{T}_{\mathrm{inv}}$  individually. For example, for XNLI,  $|\Delta_{(\mathrm{BZ - BS})}| = 19$  for the composed transformation and just 2 and 3 for  $\mathcal{T}_{\mathrm{trans}}$  and  $\mathcal{T}_{\mathrm{inv}}$  individually.  $\mathcal{T}_{\mathrm{trans}}\circ \mathcal{T}_{\mathrm{perm}}$  is worse by  $\approx 20$  points on XNLI and NER, and over 40 points on POS when compared to  $\mathcal{T}_{\mathrm{perm}}$ .  $\mathcal{T}_{\mathrm{trans}}\circ \mathcal{T}_{\mathrm{syn}}$  suffers lesser than the other two composed transformations, but it is still worse than  $\mathcal{T}_{\mathrm{syn}}$  by 3, 6, and 1 point on XNLI, NER, and POS. In conclusion, the absence of subword overlap significantly degrades zero-shot per
formance in the realistic case of languages with different word orders.
4.3 Data from the same domain boosts bilingual performance
Previously, we considered transformations  $(\mathcal{T})$  that modified the original pre-training corpus to get a parallel corpus,  $\mathcal{C}_{\mathrm{deriv}} = \mathcal{T}(\mathcal{C}_{\mathrm{orig}})$ , such that there is a one-to-one correspondence between sentences in  $\mathcal{C}_{\mathrm{orig}}$  and  $\mathcal{C}_{\mathrm{deriv}}$  (we call this setting parallel). Since procuring large parallel corpora is expensive in practice, we consider two other settings which use different corpora for original and derived.
Setup Consider two text corpora of the same size,  $\mathcal{C}_{\mathrm{orig}}^1$  and  $\mathcal{C}_{\mathrm{orig}}^2$ . We compare two settings: (1) The parallel setting pre-trains a bilingual model on  $\mathcal{C}_{\mathrm{orig}}^1 + \mathcal{T}(\mathcal{C}_{\mathrm{orig}}^1)$ , whereas the (2) non-parallel corpus setting uses  $\mathcal{C}_{\mathrm{orig}}^1 + \mathcal{T}(\mathcal{C}_{\mathrm{orig}}^2)$ . We consider two variants of non-parallel, (1) non-parallel (same) which uses different splits of Wikipedia data (hence, same domain), and (2) non-parallel (diff) which uses Wikipedia data for the original and common crawl data (web text) for the derived language (hence, different domain). We use the Transliteration transformation  $(\mathcal{T}_{\mathrm{trans}})$  to generate the derived language corpus and report  $|\Delta_{(\mathrm{BZ - BS})}|$  averaged over all languages in Figure 3.
Results We observe consistently on all tasks that the parallel setting (blue bar) performs better than both the non-parallel settings. Non-parallel (same)
Figure 2:  $|\Delta_{(\mathrm{BZ - BS})}|$  for composed transformations (§ 4.2) applied on EN as the original language. Larger scores imply worse zero-shot transfer.  $\mathcal{T}_{\mathrm{trans}} = \mathrm{Transliteration}$ ,  $\mathcal{T}_{\mathrm{inv}} = \mathrm{Inversion}$ ,  $\mathcal{T}_{\mathrm{perm}} = \mathrm{Permutation}$ , and  $\mathcal{T}_{\mathrm{syn}} = \mathrm{Syntax}$ . Sub-word overlap between the original and derived language is  $0\%$  when composed transformations are used (e.g.  $\mathcal{T}_{\mathrm{trans}} \circ \mathcal{T}_{\mathrm{inv}}$ ) and  $100\%$  when the second constituent is used (here,  $\mathcal{T}_{\mathrm{inv}}$ ). We observe that the composed transformations (green bars) do significantly worse than their constituents (blue and orange), showing that  $\mathcal{T}_{\mathrm{trans}} \circ \mathcal{T}_{\mathrm{inv}}$  is worse than  $\mathcal{T}_{\mathrm{inv}}$  by over 16 points on XNLI and 42 points on POS, with similar trends for  $\mathcal{T}_{\mathrm{trans}} \circ \mathcal{T}_{\mathrm{perm}}$ .  $\mathcal{T}_{\mathrm{trans}} \circ \mathcal{T}_{\mathrm{syn}}$  doesn't suffer as much, but its performance degradation when compared to Syntax is still large (ranges between 1 point on POS to 6 points on NER). absence of sub-word overlap significantly hurts performance when languages differ in their word orders.
performs better than non-parallel (diff), with gains ranging between 2 points on XQuAD to 17 points on NER. This result shows that even for original and derived language pairs which differ only in their script, having parallel pre-training corpora leads to the best zero-shot transfer. Since large-scale parallel unsupervised data is hard to procure, the best alternative is to use corpora from similar domains (Wikipedia) rather than different ones (Wikipedia v.s. web text).
4.4 Zero-shot performance is strongly correlated with embedding alignment
Our previous results (§ 4.2, 4.3) showed cases where zero-shot transfer between languages is poor when there is no sub-word overlap. To investigate this further, we analyze the static word embeddings learned by bilingual models and find that zero-shot transfer between languages is strongly correlated with the alignment between word embeddings for the original and derived languages.
Setup The original and the derived languages have a one-to-one correspondence between their sub-word vocabularies when we use transliteration  $(\mathcal{T}_{\mathrm{trans}})$ . For a token embedding in the original-language embedding matrix, its alignment score is  $100\%$  if it retrieves the corresponding token embedding in the derived language when a nearest-neighbor search is performed, and  $0\%$  otherwise.
We average the alignment score over all the tokens and call it alignment.
Results We measure the alignment of bilingual models pre-trained on different original-derived language pairs created using transliteration, namely the composed transformations (§ 4.2), parallel, and non-parallel (§ 4.3). We plot the alignment along with the corresponding  $\Delta_{\mathrm{(BZ - BS)}}$  scores for XNLI in Figure 4. Results for other tasks are in Appendix E.
We observe that higher alignment is associated with lower  $\Delta_{\mathrm{(BZ - BS)}}$  implying better zero-shot transfer. Alignment is lower for composed transformations like  $\mathcal{T}_{\mathrm{trans}}\circ \mathcal{T}_{\mathrm{inv}}$  and  $\mathcal{T}_{\mathrm{trans}}\circ$ $\mathcal{T}_{\mathrm{perm}}$  which have large and negative  $\Delta_{\mathrm{(BZ - BS)}}$  Alignment also explains the results in Section 4.3, with non-parallel variants having lower alignment scores than parallel, which is in line with their lower  $\Delta_{\mathrm{(BZ - BS)}}$  . Overall, we find a strong and significant Spearman's rank correlation between alignment and  $\Delta_{\mathrm{(BZ - BS)}}$  ,with  $\rho = 0.94,p <   .005$  for XNLI,  $\rho = 0.93,p <   .005$  for NER, and  $\rho = 0.89,p <   .01$  for POS, indicating that increasing the embedding alignment between languages helps improve zero-shot transfer.
Figure 3:  $|\Delta_{(\mathrm{BZ - BS})}|$  for  $\mathcal{T}_{\mathrm{trans}}$  under different conditions on the source of original and derived language pre-training corpora (hereon, corpora) ( $\S 4.3$ ), averaged over four languages. Larger values imply worse zero-shot transfer. The breakdown of scores for different languages is in Appendix D. (1) Non-parallel (diff) (green bar), which uses corpora from different domains is worse than (2) Non-parallel (same) (orange bar), which uses different sets of sentences sampled from the same domain, which is in turn worse than (3) Parallel, which uses the same sentences. Having pre-training corpora from the same domain like Wikipedia (Non-parallel (same)) gives performance boosts between 2 points for QA to 17 points for NER when compared to Non-parallel (diff).
4.5 Connections to results on natural language pairs
Effect of sub-word overlap In § 4.2, we showed that when languages have different scripts (0% subword overlap), zero-shot transfer significantly degrades when they additionally have different word orders. However, the zero-shot transfer is good when languages differ only in the script and have similar or the same word order. This is in line with anecdotal evidence in Pires et al. (2019), where zero-shot transfer works well between English and Bulgarian (different script but same subject-verb-object order - SVO), but is poor between English and Japanese (different script and word order - SVO v.s. SOV). Our result also corroborates findings in Conneau et al. (2020b) that artificially increasing sub-word overlap between natural languages (which have different word orders) improves performance (e.g., 3 points on XNLI).
Effect of token embedding alignment In  $\S 4.4$  we showed that zero-shot transfer is strongly correlated with word embedding alignment between languages. This explains the usefulness of recent stud
Figure 4:  $\Delta_{\mathrm{(BZ - BS)}}$  for Transliteration  $(\mathcal{T}_{\mathrm{trans}})$  variants on XNLI. Larger values (less negative) imply better zero-shot transfer. We see that alignment (§ 4.4) between token embeddings of different languages is correlated with  $\Delta_{\mathrm{(BZ - BS)}}$ , and hence with better zero-shot transfer. For example,  $\mathcal{T}_{\mathrm{trans}} \circ \mathcal{T}_{\mathrm{inv}}$  (bottom left) which has poor zero-shot transfer also has lower alignment, whereas Parallel (top right) which has strong transfer is accompanied with higher alignment. We find a strong and statistically significant Spearman's correlation of  $\rho_s = 0.94, p < .005$  on XNLI,  $\rho_s = 0.93, p < .005$  on NER, and  $\rho_s = 0.89, p < .01$  on POS. Plots for other tasks are in Appendix E.
ies which try to improve multilingual pre-training with the help of auxiliary objectives, which improve word or sentence embedding alignment.
DICT-MLM (Chaudhary et al., 2020) and RelateLM (Khemchandani et al., 2021) require the model to predict cross-lingual synonyms as an auxiliary objective, thus indirectly improving word-embedding alignment and the zero-shot performance on multiple tasks. Hu et al. (2021) add an auxiliary objective that implicitly improves word embedding alignment and show that they can achieve performance similar to larger models. Cao et al. (2019) explicitly improve contextual word embedding alignment with the help of word-level alignment information in machine-translated cross-lingual sentence pairs. Since they apply this post hoc and not during pre-training, the improvement, albeit significant, is small (2 points on XNLI). While these studies do not fully utilize word and sentence embedding alignment information, our results lead us to posit that they are a step in the right direction and that baking alignment information more explicitly into pre-training will be beneficial.
Through a systematic study of zero-shot transfer between four diverse natural languages and their counterparts created by modifying specific properties like the script, word order, and syntax, we showed that (1) absence of sub-word overlap hurts zero-shot performance when languages differ in their word order, and (2) zero-shot performance is strongly correlated with word embedding alignment between languages. Some recent studies have implicitly or unknowingly attempted to improve alignment and have shown slight improvements in zero-shot transfer performance. However, our results lead us to posit that explicitly improving word embedding alignment during pre-training by using either supervised (e.g., parallel sentences and translation dictionaries) or unsupervised data will significantly improve zero-shot transfer. Although real languages typically differ in more ways than the set of properties considered in our transformations, our methodology is still useful to help isolate crucial properties for transfer. Future work can experiment with more sophisticated transformations and investigate closer connections with human language pairs.
This work was funded through a grant from the Chadha Center for Global India at Princeton University. We thank Shunyu Yao, Vishvak Murahari, Tianyu Gao, Sadhika Malladi, and Jens Tuyls for reading our draft and providing valuable comments. We also thank the Google Cloud Research program for computational support in running our experiments.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020a. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 4623-4637. Association for Computational Linguistics.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020b. On the cross-lingual transferability of monolingual representations. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Samuel Bowman, Christopher Potts, and Christopher D Manning. 2015. Recursive neural networks can
learn logical semantics. In Proceedings of the 3rd workshop on continuous vector space models and their compositionality, pages 12-21.
Steven Cao, Nikita Kitaev, and Dan Klein. 2019. Multilingual alignment of contextual word representations. In International Conference on Learning Representations.
Aditi Chaudhary, Karthik Raman, Krishna Srinivasan, and Jiecao Chen. 2020. DICT-MLM: improved multilingual pre-training using bilingual dictionaries. CoRR, abs/2010.12566.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020a. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 8440-8451. Association for Computational Linguistics.
Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 7057-7067.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating crosslingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b. Emerging cross-lingual structure in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6022–6034.
Jacob Devlin. 2019. Multilingual-BERT. https://github.com/google-research/bert/blob/master/multilingual.md. [Online; accessed 20-December-2020].
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Sumanth Doddapaneni, Gowtham Ramesh, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M Khapra. 2021. A primer on pretrained multilingual language models. arXiv preprint arXiv:2107.00676.
Philipp Dufter and Hinrich Schütze. 2020. Identifying elements essential for bert's multilinguality. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4423-4437.
Tsung-Yuan Hsu, Chi-Liang Liu, and Hung-Yi Lee. 2019. Zero-shot reading comprehension by cross-lingual transfer learning with multi-lingual language representation model. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5933-5940.
Junjie Hu, Melvin Johnson, Orhan First, Aditya Siddhant, and Graham Neubig. 2021. Explicit alignment objectives for multilingual bidirectional encoders. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3633-3643.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. Xtreme: A massively multilingual multitask benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411-4421. PMLR.
Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual BERT: an empirical study. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, NC Gokul, Avik Bhattacharyya, Mitesh M Khapra, and Pratyush Kumar. 2020. inlpsuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for indian languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 4948-4961.
Yash Khemchandani, Sarvesh Mehtani, Vaidehi Patil, Abhijeet Awasthi, Partha Talukdar, and Sunita Sarawagi. 2021. Exploiting language relatedness for low web-resource language model adaptation: An Indic languages study. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1312-1323, Online. Association for Computational Linguistics.
Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, et al. 2019. Choosing transfer languages for cross-lingual learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3125-3135.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843.
Joakim Nivre, Mitchell Abrams, Željko Agić, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, and Mohammed Attia et al. 2018. Universal dependencies 2.2. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Crosslingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946-1958.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101-108.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations.
Dingquan Wang and Jason Eisner. 2016. The galactic dependencies treebanks: Getting more data by synthesizing new languages. Transactions of the Association for Computational Linguistics, 4:491-505.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 833-844. Association for Computational Linguistics.
Zhengxuan Wu, Isabel Papadimitriou, and Alex Tamkin. 2022. Oolong: Investigating what makes crosslingual transfer hard with controlled studies. arXiv preprint arXiv:2202.12312.
A Mathematical Specification for Transformation of Downstream Datasets
Text classification Text classification tasks like news classification or sentiment analysis typically have instances which contain a single sentence and a label. Instances in other classification tasks like natural language inference (NLI) (Bowman et al., 2015) contain two sentences and one label. For such tasks, we apply the transformation  $(\mathcal{T})$  on each sentence within every instance, and leave the annotated label as is. Therefore, for a dataset of size  $n$  which contains  $m$  sentences per instance, we have:
$$
\mathcal {D} _ {\mathrm {o r i g}} = \left\{\left(s _ {i 1}, \ldots , s _ {i m}, y _ {i}\right) \mid i = 1: N \right\}
$$
$$
\mathcal {D} _ {\text {d e r i v}} = \left\{\left(\mathcal {T} \left(s _ {i 1}\right), \dots , \mathcal {T} \left(s _ {i m}\right), y _ {i}\right) \mid i = 1: N \right\}
$$
Token-classification tasks Tasks like namedentity recognition (NER) and part-of-speech tagging (POS tagging) have labels associated with each token in the sentence. For these datasets, we ensure that any transformation  $(\mathcal{T})$  that changes the order of the tokens also changes the order of the corresponding labels.