SlowGuess/NAACL_2021_OCR huggingface dataset:
Ablation Table 5 shows the contribution of each component of our method on XNLI. Removing TLM (-TLM) consistently leads to about  $1\%$  accuracy drop across the board, showing positive effects of the word-alignment objective. To better understand TLM's consistent improvement, we replace TLM with MLM (repl TLM w/ MLM), where we treat  $S_{i}^{en}$  and  $S_{i}^{str}$  from the parallel corpora as separate monolingual sequences and perform MLM on each of them. The masking scheme is the same
as TLM described in Section 2. We observe that MLM does not bring significant improvement. This confirms that the improvement of TLM is not from the encoders being trained with more data and iterations. Instead, the word-alignment nature of TLM does help the multilingual training.
Comparing our model without word-level alignment, i.e., -TLM, to the baseline mBERT in Table 2, we get  $2 - 4\%$  improvement in the zero-shot setting and  $1 - 2\%$  improvement in translate-train as the amount of parallel data is increased. These are relatively large improvements considering the fact that only sentence-level alignment is used. This also conforms to our intuition that sentence-level alignment is a good fit here since XNLI is a sentence-level task.
In the zero-shot setting, removing MoCo (-MoCo) performs similarly to -TLM, where we observe an accuracy drop of about  $1\%$  compared to our full system. In translate-train, -MoCo outperforms -TLM and even matches the full system performance for 250k.
Finally, we show ablation result for our code-switching in translate-train. On average, code-switching provides an additional gain of  $1\%$ .
Training Multilingual LMs with Shared Vocabulary mBERT (Devlin et al., 2019) is trained us
Model	en	ar	de	es	hi	zh	avg
Zero-shot cross-lingual transfer
mBERT from (Lewis et al., 2020)	77.7	45.7	57.9	64.3	43.8	57.5	57.8
mBERT from (Hu et al., 2020)	80.2	52.3	59.0	67.4	50.2	59.6	61.5
Ours (250k)	80.0	52.6	63.2	67.7	54.1	60.5	63.0
Ours (600k)	79.7	52.4	62.8	67.6	58.3	60.4	63.5
Ours (2M)	79.8	53.8	62.3	67.7	57.9	61.5	63.8
XLM from (Lewis et al., 2020)	74.9	54.8	62.2	68.0	48.8	61.1	61.6
XLM-RBase (Conneau et al., 2020)	77.1	54.9	60.9	67.4	59.4	61.8	63.6
Translate-train
mBERT from (Lewis et al., 2020)	77.7	51.8	62.0	53.9	55.0	61.4	60.3
mBERT from (Hu et al., 2020)	80.2	55.0	64.6	70.0	60.1	63.9	65.6
Ours (250k)	80.0	58.0	65.7	71.0	62.0	64.4	66.9
Ours (600k)	79.7	58.1	65.2	70.5	63.4	64.1	66.8
Ours (2M)	79.8	58.2	64.7	70.6	63.1	64.4	66.8
XLM from (Lewis et al., 2020)	74.9	54.0	61.4	65.2	50.7	59.8	61.0
Table 4: MLQA F1 scores for each language. After alignment, our best model improves over mBERT baseline by  $2.3\%$  and outperforms XLM- $\mathbf{R_{Base}}$  for zero-shot transfer. Our model trained with the smallest amount of parallel data is  $1.3\%$  better than mBERT baseline for translate-train.
ing MLM and NSP objectives on Wikipedia data in 104 languages with a shared vocabulary. Several works study what makes this pretrained model multilingual, and why it works well for cross-lingual transfer. Pires et al. (2020) hypothesize that having a shared vocabulary for all languages helps mapping tokens to a shared space. However, K et al. (2020) train several bilingual BERT models such as en-es, and enface-es, where data for enface is constructed by Unicode shifting of the English data such that there is no character overlap with data of the other language. Result shows that enface-es still transfers well to Spanish and the contribution from shared vocabulary is very small. The authors point out that model depth and capacity instead are the key factors contributing to mBERT's cross-lingual transferability. XLM-R (Conneau et al., 2020) improves over mBERT by training longer with more data from CommonCrawl, and without the NSP objective. In terms of model size, XLM-R uses over 3x more parameters than mBERT. Its base version, XLM- $\mathbf{R}_{\mathrm{Base}}$ , is more comparable to mBERT with the same hidden size and number of attention heads, but a larger shared vocabulary.
Training Multilingual LMs with Parallel Sentences In addition to MLM on monolingual data, XLM (Conneau and Lample, 2019) further improves their cross-lingual LM pretraining by introducing a new TLM objective on parallel data. TLM concatenates source and target sentences together, and predicts randomly masked tokens. Our work uses a slightly different version of TLM to
together with a contrastive objective to post-pretrain mBERT. Unlike XLM, our TLM does not reset positions of target sentences, and does not use language embeddings. We also randomly shuffle the order of source and target sentences. Another difference between XLM and our work is XLM has  $45\%$  more parameters and uses more training data. Similar to XLM, Unicoder (Huang et al., 2019) pretrains LMs on multilingual corpora. In addition to MLM and TLM, they introduce three additional cross-lingual pretraining tasks: word recover, paraphrase classification, and mask language model. Yang et al. (2020) propose Alternating Language Modeling (ALM). On a pair of bilingual sequences, instead of TLM, they perform phrase-level code-switching and MLM on the code-switched sequence. ALM is pretrained on both monolingual Wikipedia data and 1.5B code-switched sentences.
Training mBERT with Word Alignments Cao et al. (2020) post-align mBERT embeddings by first generating word alignments on parallel sentences that involve English. For each aligned word pair, the  $L_{2}$  distance between their embeddings is minimized to train the model. In order to maintain original transferability to downstream tasks, a regularization term is added to prevent the target language embeddings from deviating too much from their mBERT initialization. Our approach post-aligns mBERT with two self-supervised signals from parallel data without using pre-alignment tools. Wang et al. (2019) also align mBERT em
Model	en	fr	es	de	bg	ar	zh	hi	avg
Zero-shot cross-lingual transfer
Our full system (250k)	82.4	75.5	76.2	73.3	74.6	68.2	71.7	62.8	73.1
- MoCo	82.2	75.3	75.8	73.0	71.3	67.1	71.3	61.8	72.2
- TLM	80.5	74.7	75.2	71.4	72.7	66.2	68.9	64.0	71.7
repl TLM w/ MLM	81.5	75.0	75.2	70.8	72.5	66.2	69.0	61.9	71.5
Our full system (600k)	82.4	76.7	76.4	74.0	74.1	69.1	72.3	66.9	74.0
- MoCo	82.0	75.5	75.9	72.8	72.1	68.5	72.1	64.5	72.9
- TLM	81.2	75.1	75.4	71.9	73.3	68.2	71.0	65.8	72.7
repl TLM w/ MLM	82.2	75.7	75.5	73.0	73.3	68.5	71.1	66.5	73.2
Our full system (2M)	82.8	76.6	76.7	74.2	73.8	70.3	72.8	66.9	74.3
- MoCo	82.5	75.2	76.3	72.4	71.9	67.9	71.4	65.2	72.9
- TLM	81.3	76.2	76.4	73.2	72.9	69.0	71.5	66.1	73.3
repl TLM w/ MLM	82.0	75.8	75.8	73.2	73.5	68.7	70.6	65.8	73.2
Translate-train
Our full system (250k)	82.4	78.8	79.0	78.7	78.4	74.0	77.9	69.6	77.4
- MoCo	82.2	79.8	79.8	77.8	78.9	73.8	77.3	69.8	77.4
- TLM	80.5	78.3	77.8	77.5	77.4	72.4	77.2	69.5	76.3
repl TLM w/ MLM	81.5	78.4	79.4	78.3	78.2	73.4	76.9	69.9	77.0
- CS	82.4	77.8	79.5	76.2	76.2	73.2	77.5	67.9	76.3
Our full system (600k)	82.4	79.7	79.7	77.9	79.0	75.2	77.8	71.5	77.9
- MoCo	82.0	79.5	79.2	78.1	78.9	74.1	78.1	71.0	77.6
- TLM	81.2	78.5	78.6	78.1	77.7	73.7	76.6	70.8	76.9
repl TLM w/ MLM	82.2	78.4	78.4	77.1	78.0	73.9	76.9	70.8	77.0
- CS	82.4	79.2	78.3	77.5	77.0	73.6	77.3	69.9	76.9
Our full system (2M)	82.8	79.7	80.6	78.6	78.8	75.2	78.0	72.0	78.2
- MoCo	82.5	79.1	80.0	79.1	78.5	75.3	77.7	70.5	77.8
- TLM	81.3	78.9	79.4	78.0	77.8	74.4	77.2	70.0	77.1
repl TLM w/ MLM	82.0	79.1	79.0	78.2	77.8	74.3	77.7	70.4	77.3
- CS	82.8	79.1	79.0	78.0	77.5	73.6	77.1	69.5	77.1
Table 5: Ablation Study on XNLI. 250k, 600k, 2M refer to the maximum number of parallel sentence pairs per language used in PPA. MoCo refers to our sentence-level alignment task using contrastive learning. TLM refers to our word-level alignment task with translation language modeling. CS stands for code-switching. We conduct an additional study repl TLM w/ MLM, which means instead of TLM training, we augment our sentence-level alignment with regular MLM on monolingual text. This ablation confirms that the TLM objective helps because of its word alignment capability, not because we train the encoders with more data and iterations.
beddings using parallel data. They learn a linear transformation that maps a word embedding in a target language to the embedding of the aligned word in the source language. They show that their transformed embeddings are more effective on zero-shot cross-lingual dependency parsing.
Besides the aforementioned three major directions, Artetxe and Schwenk (2019) train a multilingual sentence encoder on 93 languages. Their stacked BiLSTM encoder is trained by first generating embedding of a source sentence and then decoding the embedding into the target sentence in other languages.
Concurrent to our work, Chi et al. (2020), Feng et al. (2020) and Wei et al. (2020) also leverage variants of contrastive learning for cross-lingual alignment. We focus on a smaller model and improve on it using as little parallel data as possible. We also explore code-switching during finetuning on downstream tasks to complement the post-pretraining
alignment objectives.
Post-pretraining embedding alignment is an efficient means of improving cross-lingual transferability of pretrained multilingual LMs, especially when pretraining from scratch is not feasible. We showed that our self-supervised sentence-level and word-level alignment tasks can greatly improve mBERT's performance on downstream tasks of NLI and QA, and the method can potentially be applied to improve other pretrained multilingual LMs.
In addition to zero-shot cross-lingual transfer, we also showed that code-switching with English during finetuning provides additional alignment signals, when training data is available for the target language.
Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics (TACL), 7:597-610.
Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Multilingual alignment of contextual word representations. In Proceedings of the 8th International Conference on Learning Representation (ICLR), Addis Ababa, Ethiopia.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, pages 1-18.
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2020. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. arXiv preprint arXiv:2007.07834, pages 1-11.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishray Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 8440-8451, Seattle.
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In Advances in Neural Information Processing Systems (NIPS), pages 7059-7069, Vancouver.
Alexis Conneau, Guillaume Lample, Rudy Rinott, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2475–2485, Brussels.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In *Processing of the IEEE Conference on Computer Vision and Pattern Recognition* (CVPR), pages 248–255, Miami.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 20th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4171-4186, Minneapolis. The Association for Computational Linguistics.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 644-648, Atlanta. The Association for Computational Linguistics.
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic BERT sentence embedding. arXiv preprint arXiv:2007.01852, pages 1-13.
Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1735-1742, New York.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9726-9735, Seattle.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalization. arXiv preprint arXiv:2003.11080, pages 1-20.
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. Unicoder: A universal language encoder by pretraining with multiple cross-lingual tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2485-2494, Hong Kong.
Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual BERT: An empirical study. In Proceedings of the 8th International Conference on Learning Representation (ICLR), Addis Ababa, Ethiopia.
Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. 2018. The IIT Bombay English-Hindi parallel corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC), pages 3473-3476, Miyazaki, Japan.
Patrick Lewis, Barlas Oğuz, Rudy Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1-16, Seattle.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis,
Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, pages 1-13.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2020. How multilingual is multilingual BERT? arXiv preprint arXiv:1906.01502, pages 1-6.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD:  $100,000+$  questions for machine comprehension of text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2383-2392, Austin, Texas.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC), pages 2214-2218, Istanbul.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, pages 1-13.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS), pages 5998-6008, Long Beach, CA.
Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. 2019. Cross-lingual BERT transformation for zero-shot dependency parsing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5721-5727, Hong Kong.
Xiangpeng Wei, Yue Hu, Rongxiang Weng, Luxi Xing, Heng Yu, and Weihua Luo. 2020. On learning universal representations across languages. arXiv preprint arXiv:2007.15960, pages 1-13.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 19th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL), pages 1112-1122, New Orleans. The Association for Computational Linguistics.
Shijie Wu and Mark Dredze. 2019. Beto, Bentz, Becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844, Hong Kong.
Jian Yang, Shuming Ma, Dongdong Zhang, ShuangZhi Wu, Zhoujun Li, and Ming Zhou. 2020. Alternating language modeling for cross-lingual pre-training.
In Proceedings of the 34th Conference on Artificial Intelligence (AAAI), pages 9386-9393, New York. AAAI Press.
Michal Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The United Nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC), pages 3530-3534, Portoorž, Slovenia.