SlowGuess/NAACL_2021_OCR huggingface dataset:
understanding and generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6008-6018, Online. Association for Computational Linguistics.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8(0):726-742.
Xuanlong Nguyen, Martin J Wainwright, and Michael I Jordan. 2010. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11):5847-5861.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001, Florence, Italy. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Masoud Jalili Sabet, Philipp Dufter, and Hinrich Schütze. 2020. Simalign: High quality word alignments without parallel training data using static and contextualized embeddings. arXiv preprint arXiv:2004.08728.
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019. WikiMatrix: Mining 135M parallel sentences in 1620 language pairs from wikipedia. arXiv preprint arXiv:1907.05791.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. MASS: Masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5926-5936. PMLR.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2019. Contrastive multiview coding. ArXiv, abs/1906.05849.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation, pages 2214-2218, Istanbul, Turkey. European Language Resources Association.
Naftali Tishby and Noga Zaslavsky. 2015. Deep learning and the information bottleneck principle. 2015 IEEE Information Theory Workshop (ITW), pages 1-5.
T. Tsujishita. 1995. On triple mutual information. Advances in Applied Mathematics, 16(3):269 - 274.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008. Curran Associates, Inc.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 833-844, Hong Kong, China. Association for Computational Linguistics.
Jian Yang, Shuming Ma, Dongdong Zhang, Shuangzhi Wu, Zhoujun Li, and Ming Zhou. 2020. Alternating language modeling for cross-lingual pre-training. In Thirty-Fourth AAAI Conference on Artificial Intelligence.
Michal Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus v1.0. In LREC, pages 3530-3534.
A Pre-Training Data
We reconstruct CCNet $^2$  and follow (Conneau et al., 2020a) to reproduce the CC-100 corpus for monolingual texts. The resulting corpus contains 94 languages. Table 8 reports the language codes and data size in our work. Notice that several languages can share the same ISO language code, e.g., zh represents both Simplified Chinese and Traditional Chinese. Moreover, Table 9 shows the statistics of the parallel data.
Code	Size (GB)	Code	Size (GB)	Code	Size (GB)
af	0.2	hr	1.4	pa	0.8
am	0.4	hu	9.5	pl	28.6
ar	16.1	hy	0.7	ps	0.4
as	0.1	id	17.2	pt	39.4
az	0.8	is	0.5	ro	11.0
ba	0.2	it	47.2	ru	253.3
be	0.5	ja	86.8	sa	0.2
bg	7.0	ka	1.0	sd	0.2
bn	5.5	kk	0.6	si	1.3
ca	3.0	km	0.2	sk	13.6
ckb	0.6	kn	0.3	sl	6.2
cs	14.9	ko	40.0	sq	3.0
cy	0.4	ky	0.5	sr	7.2
da	6.9	la	0.3	sv	60.4
de	99.0	lo	0.2	sw	0.3
el	13.1	lt	2.3	ta	7.9
en	731.6	lv	1.3	te	2.3
eo	0.5	mk	0.6	tg	0.7
es	85.6	ml	1.3	th	33.0
et	1.4	mn	0.4	tl	1.2
eu	1.0	mr	0.5	tr	56.4
fa	19.0	ms	0.7	tt	0.6
fi	5.9	mt	0.2	ug	0.2
fr	89.9	my	0.4	uk	13.4
ga	0.2	ne	0.6	ur	3.0
gl	1.5	nl	25.9	uz	0.1
gu	0.3	nn	0.4	vi	74.5
he	4.4	no	5.5	yi	0.3
hi	5.0	or	0.3	zh	96.8
Table 8: The statistics of CCNet used corpus for pretraining.
ISO Code	Size (GB)	ISO Code	Size (GB)
en-ar	5.88	en-ru	7.72
en-bg	0.49	en-sw	0.06
en-de	4.21	en-th	0.47
en-el	2.28	en-tr	0.34
en-es	7.09	en-ur	0.39
en-fr	7.63	en-vi	0.86
en-hi	0.62	en-zh	4.02
B Results of Training From Scratch
We conduct experiments under the setting of training from scratch. The Transformer size and hyperparameters follow BERT-base (Devlin et al., 2019). The parameters are randomly initialized from  $U[-0.02, 0.02]$ . We optimize the models with
Table 9: Parallel data used for pre-training.
Model Metrics	XNLI Acc.	MLQA F1 / EM
MMLMSCRATCH	69.40	55.02 / 37.90
INFOXLMSCRATCH	70.71	59.71 / 41.46
-XLCo	70.64	57.70 / 40.21
-TLM	69.76	58.22 / 40.78
-MMLM	63.06	52.81 / 35.01
Adam using a batch size of 256 for a total of 1M steps. The learning rate is scheduled with a linear decay with 10K warmup steps, where the peak learning rate is set as 0.0001. For cross-lingual contrast, we set the queue length as 16,384. We use a warmup of 200K steps for the key encoder and then enable cross-lingual contrast. We use an inverse square root scheduler to set the momentum coefficient, i.e.,  $m = \min (1 - t^{-0.51},0.9995)$ , where  $t$  is training step.
Table 10 shows the results of INFOXLM $_{\text{SCRATCH}}$  and various ablations. INFOXLM $_{\text{SCRATCH}}$  significantly outperforms MMLM $_{\text{SCRATCH}}$  on both XNLI and MLQA. We also evaluate the pre-training objectives of INFOXLM, where we ablate XLCo, TLM, and MMLM, respectively. The findings agree with the results in Table 7.
C Hyperparameters for Pre-Training
As shown in Table 11, we present the hyperparameters for pre-training INFOXLM. We use the same vocabulary with XLM-R (Conneau et al., 2020a).
Table 10: Ablation results of the models pre-trained from scratch. Results are averaged over five runs.
Hyperparameters	FROM SCRATCH	BASE	LARGE
Layers	12	12	24
Hidden size	768	768	1,024
FFN inner hidden size	3,072	3,072	4,096
Attention heads	12	12	16
Training steps	1M	150K	200K
Batch size	256	2,048	2,048
Adam ε	1e-6	1e-6	1e-6
Adam β	(0.9, 0.999)	(0.9, 0.98)	(0.9, 0.98)
Learning rate	1e-4	2e-4	1e-4
Learning rate schedule	Linear	Linear	Linear
Warmup steps	10,000	10,000	10,000
Gradient clipping	1.0	1.0	1.0
Weight decay	0.01	0.01	0.01
Momentum coefficient	0.9995*	0.9999	0.999
Queue length	16,384	131,072	131,072
Universal layer	8	8	12
Table 11: Hyperparameters used for INFOXLM pretraining. \*: the momentum coefficient uses an inverse square root scheduler  $m = \min (1 - t^{-0.51}$  , 0.9995).
XNLI	MLQA
Batch size	32	{16, 32}
Learning rate	{5e-6, 7e-6, 1e-5}	{2e-5, 3e-5, 5e-5}
LR schedule	Linear	Linear
Warmup	12,500 steps	10%
Weight decay	0	0
Epochs	10	{2, 3, 4}
Table 12: Hyperparameters used for fine-tuning BASE-size models on XNLI and MLQA.
XNLI	MLQA
Batch size	32	32
Learning rate	{4e-6, 5e-6, 6e-6}	{2e-5, 3e-5, 5e-5}
LR schedule	Linear	Linear
Warmup	5,000 steps	10%
Weight decay	{0, 0.01}	0
Epochs	10	{2, 3, 4}
Table 13: Hyperparameters used for fine-tuning LARGE-size models on XNLI and MLQA.
D Hyperparameters for Fine-Tuning
In Table 12 and Table 13, we present the hyperparameters for fine-tuning on XNLI and MLQA. For each task, the hyperparameters are searched on the joint validation set of all languages (\#M=1). For XNLI, we evaluate the model every 5,000 steps, and select the model with the best accuracy score on the validation set. For MLQA, we directly use the final learned model. The final scores are averaged over five random seeds.