SlowGuess/NAACL_2021_OCR huggingface dataset:
Models	#M	en	fr	es	de	el	bg	ru	tr	ar	vi	th	zh	hi	sw	ur	Avg
Fine-tune multilingual model on English training set (Cross-lingual Transfer)
MBERT*	N	82.1	73.8	74.3	71.1	66.4	68.9	69.0	61.6	64.9	69.5	55.8	69.3	60.0	50.4	58.0	66.3
XLM (w/o TLM)*	N	83.7	76.2	76.6	73.7	72.4	73.0	72.1	68.1	68.4	72.0	68.2	71.5	64.5	58.0	62.4	71.3
XLM*	N	85.0	78.7	78.9	77.8	76.6	77.4	75.3	72.5	73.1	76.1	73.2	76.5	69.6	68.4	67.3	75.1
XLM (w/o TLM)*	1	83.2	76.7	77.7	74.0	72.7	74.1	72.7	68.7	68.6	72.9	68.9	72.5	65.6	58.2	62.4	70.7
UNICODER	1	85.4	79.2	79.8	78.2	77.3	78.5	76.7	73.8	73.9	75.9	71.8	74.7	70.1	67.4	66.3	75.3
XLM-R*	1	85.8	79.7	80.7	78.7	77.5	79.6	78.1	74.2	73.8	76.5	74.6	76.7	72.4	66.5	68.3	76.2
XLM-R (reimpl)	1	84.7	79.1	79.4	77.4	76.6	78.4	76.0	73.5	72.6	75.5	73.0	74.5	71.0	65.7	67.6	75.0
INFOXLM	1	86.4	80.3	80.9	79.3	77.8	79.3	77.6	75.6	74.2	77.1	74.6	77.0	72.2	67.5	67.3	76.5
-XLCo	1	86.5	80.5	80.3	78.7	77.3	78.8	77.4	74.6	73.8	76.8	73.7	76.7	71.8	66.3	66.4	76.0
XLM-RLARGE*	1	89.1	84.1	85.1	83.9	82.9	84.0	81.2	79.6	79.8	80.8	78.1	80.2	76.9	73.9	73.8	80.9
XLM-RLARGE (reimpl)	1	88.9	83.6	84.8	83.1	82.4	83.7	80.7	79.2	79.0	80.4	77.8	79.8	76.8	72.7	73.3	80.4
INFOXLM LARGE	1	89.7	84.5	85.5	84.1	83.4	84.2	81.3	80.9	80.4	80.8	78.9	80.9	77.9	74.8	73.7	81.4
Fine-tune multilingual model on all training sets (Translate-Train-All)
XLM (w/o TLM)*	1	84.5	80.1	81.3	79.3	78.6	79.4	77.5	75.2	75.6	78.3	75.7	78.3	72.1	69.2	67.7	76.9
XLM*	1	85.0	80.8	81.3	80.3	79.1	80.9	78.3	75.6	77.6	78.5	76.0	79.5	72.9	72.8	68.5	77.8
XLM-R*	1	85.4	81.4	82.2	80.3	80.4	81.3	79.7	78.6	77.3	79.7	77.9	80.2	76.1	73.1	73.0	79.1
XLM-R (reimpl)	1	85.0	81.0	81.9	80.6	79.7	81.4	79.5	77.7	77.3	79.5	77.5	79.1	75.3	72.2	70.9	78.6
INFOXLM	1	86.5	82.6	83.0	82.3	81.3	82.4	80.6	79.5	78.9	81.0	78.9	80.7	77.8	73.3	71.6	80.0
Table 1: Evaluation results on XNLI cross-lingual natural language inference. We report test accuracy in 15 languages. The model number  $\# \mathrm{M} = \mathrm{N}$  indicates the model selection is done on each language's validation set (i.e., each language has a different model), while  $\# \mathrm{M} = 1$  means only one model is used for all languages. Results with “*” are taken from Conneau et al. (2020a). “(reimpl)” is our reimplementation of fine-tuning, which is the same as INFOXLM. Results of INFOXLM and XLM-R (reimpl) are averaged over five runs. “-XLCo” is the model without cross-lingual contrast.
languages; (3) XLM-R (Conneau et al., 2020a) scales up MMLM to the large CC-100 corpus in 100 languages with much more training steps; (4) UNICODER (Liang et al., 2020) continues training XLM-R with MMLM and TLM. (5) INFOXL - XLCO continues training XLM-R with MMLM and TLM, using the same pre-training datasets with INFOXLM.
Cross-Lingual Natural Language Inference Table 1 reports the classification accuracy on each test of XNLI under the above evaluation settings. The final scores on test set are averaged over five random seeds. INFOXLM outperforms all baseline models on the two evaluation settings of XNLI. In the cross-lingual transfer setting, INFOXLM achieves 76.5 averaged accuracy, outperforming XLM-R (reimpl) by 1.5. Similar improvements can be observed for large-size models. Moreover, the ablation results “—XLCo” show that cross-lingual contrast is helpful for zero-shot transfer in most languages. We also find that INFOXLM improves the results in the translate-train-all setting.
Cross-Lingual Sentence Retrieval In Table 2 and Table 3, we report the top-1 accuracy scores of cross-lingual sentence retrieval with the base-size
models. The evaluation results demonstrate that INFOXLM produces better aligned cross-lingual sentence representations. On the 14 language pairs that are covered by parallel data, INFOXLM obtains 77.8 and 80.6 averaged top-1 accuracies in the directions of xx  $\rightarrow$  en and en  $\rightarrow$  xx, outperforming XLM-R by 20.2 and 21.1. Even on the 22 language pairs that are not covered by parallel data, INFOXLM outperforms XLM-R on 16 out of 22 language pairs, providing  $8.1\%$  improvement in averaged accuracy. In comparison, the ablation variant “-XLCo” (i.e., MMLM+TLM) obtains better results than XLM-R in Table 2, while getting worse performance than XLM-R in Table 3. The results indicate that XLCo encourages the model to learn universal representations even on the language pairs without parallel supervision.
Cross-Lingual Question Answering Table 4 compares INFOXLM with baseline models on MLQA, where we report the F1 and the exact match (EM) scores on each test set. Both INFOXLM and INFOXLMLarge obtain the best results against the four baselines. In addition, the results of the ablation variant “-XLCo” indicate that the proposed cross-lingual contrast is beneficial on MLQA.
Models	Direction	ar	bg	zh	de	el	fr	hi	ru	es	sw	th	tr	ur	vi	Avg
XLM-R	xx → en	36.8	67.6	60.7	89.9	53.7	74.1	54.2	72.5	74.0	18.7	38.3	61.1	36.6	68.4	57.6
INFOXLM	xx → en	59.0	78.6	86.3	93.9	62.1	79.4	87.1	83.8	88.2	39.5	84.9	83.3	73.0	89.6	77.8
-XLCo	xx → en	42.9	65.5	69.5	91.1	55.6	76.4	71.6	74.9	74.8	20.5	68.1	69.8	51.6	81.8	65.3
XLM-R	en → xx	38.6	69.9	60.3	89.4	57.3	74.3	49.3	73.0	74.6	14.4	58.4	64.0	36.9	72.5	59.5
INFOXLM	en → xx	68.6	78.6	86.4	95.1	72.6	84.0	88.3	85.7	87.2	40.8	91.2	84.7	73.3	92.0	80.6
-XLCo	en → xx	45.4	64.0	69.3	88.1	56.5	72.3	69.6	73.6	71.5	22.1	79.7	64.3	48.2	79.8	64.6
Table 2: Evaluation results on Tatoeba cross-lingual sentence retrieval. We report the top-1 accuracy of 14 language pairs that are covered by parallel data.
Models	Direction	af	bn	et	eu	fi	he	hu	id	it	jv	ja	ka	kk	ko	ml	mr	nl	fa	pt	ta	te	tl	Avg
XLM-R	xx → en	55.2	29.3	49.3	33.5	66.7	53.9	61.6	70.8	68.2	15.1	57.2	41.4	40.3	51.6	56.5	46.0	79.5	68.0	80.6	25.7	32.5	31.2	50.6
INFOXLM	xx → en	48.6	49.6	38.3	36.7	65.7	62.9	61.7	79.9	72.2	13.2	78.3	57.4	49.2	74.5	76.6	72.0	80.8	82.2	84.7	53.7	53.0	42.1	60.6
-XLCo	xx → en	33.1	33.5	25.9	20.8	48.4	49.1	46.1	68.5	60.4	12.2	60.6	38.6	35.1	60.6	57.8	49.1	72.2	66.0	75.3	36.5	38.0	25.5	46.1
XLM-R	en → xx	55.0	27.9	50.2	32.5	72.9	63.2	67.1	71.9	68.0	9.8	58.2	52.0	41.7	58.3	60.8	42.1	78.9	69.6	82.1	33.2	38.9	29.7	52.9
INFOXLM	en → xx	51.8	49.1	35.2	28.6	65.6	66.5	61.7	80.1	72.8	7.8	80.4	61.9	50.6	79.6	78.7	68.1	81.8	82.8	86.5	63.5	53.0	35.5	61.0
-XLCo	en → xx	28.1	23.5	19.0	12.6	45.2	49.7	40.8	62.8	57.5	3.4	58.2	38.9	31.3	61.0	57.5	37.2	67.8	66.4	75.0	43.0	31.6	17.9	42.2
Table 3: Evaluation results on Tatoeba cross-lingual sentence retrieval. We report the top-1 accuracy scores of 22 language pairs that are not covered by parallel data.
To understand INFOXLM and the cross-lingual contrast task more deeply, we conduct analysis from the perspectives of cross-lingual transfer and cross-lingual representations. Furthermore, we perform comprehensive ablation studies on the major components of INFOXLM, including the cross-lingual pre-training tasks, mixup contrast, the contrast layer, and the momentum contrast. To reduce the computation load, we use INFOXLM15 in our ablation studies, which is trained on 15 languages for 100K steps.
Cross-Linguual Transfer Gap Cross-lingual transfer gap (Hu et al., 2020) is the difference between the performance on the English test set and the averaged performance on the test sets of all other languages. A lower cross-lingual transfer gap score indicates more end-task knowledge from the English training set is transferred to other languages. In Table 5, we compare the cross-lingual transfer gap scores of INFOXLM with baseline models on MLQA and XNLI. Note that we do not include the results of XLM because it is pre-trained on 15 languages or using #M=N. The results show that INFOXLM reduces the gap scores on both MLQA and XNLI, providing better cross-lingual transferability than the baselines.
Cross-Lingual Representations In addition to cross-lingual transfer, learning good cross-lingual representations is also the goal of cross-lingual pre
Figure 1: Evaluation results of different layers on Tatoeba cross-lingual sentence retrieval.
training. In order to analyze how the cross-lingual contrast task affects the alignment of the learned cross-lingual representations, we evaluate the representations of different middle layers on the Tatoeba test sets of the 14 languages that are covered by parallel data. Figure 1 presents the averaged top-1 accuracy of cross-lingual sentence retrieval in the direction of xx  $\rightarrow$  en. INFOXLM outperforms XLM-R on all of the 12 layers, demonstrating that our proposed task improves the cross-lingual alignment of the learned representations. From the results of XLM-R, we observe that the model suffers from a performance drop in the last few layers. The reason is that MMLM encourages the representations of the last hidden layer to be similar to token embeddings, which is contradictory with the goal of learning cross-lingual representations. In
Models	en	es	de	ar	hi	vi	zh	Avg
MBERT*	77.7 / 65.2	64.3 / 46.6	57.9 / 44.3	45.7 / 29.8	43.8 / 29.7	57.1 / 38.6	57.5 / 37.3	57.7 / 41.6
XLM*	74.9 / 62.4	68.0 / 49.8	62.2 / 47.6	54.8 / 36.3	48.8 / 27.3	61.4 / 41.8	61.1 / 39.6	61.6 / 43.5
UNICODER	80.6 / -	68.6 / -	62.7 / -	57.8 / -	62.7 / -	67.5 / -	62.1 / -	66.0 / -
XLM-R	77.1 / 64.6	67.4 / 49.6	60.9 / 46.7	54.9 / 36.6	59.4 / 42.9	64.5 / 44.7	61.8 / 39.3	63.7 / 46.3
XLM-R (reimpl)	80.2 / 67.0	67.7 / 49.9	62.1 / 47.7	56.1 / 37.2	61.1 / 44.0	67.0 / 46.3	61.4 / 38.5	65.1 / 47.2
INFOXLM	81.6 / 68.3	69.8 / 51.6	64.3 / 49.4	60.6 / 40.9	65.2 / 47.1	70.2 / 49.0	64.8 / 41.3	68.1 / 49.6
-XLCo	81.2 / 68.1	69.6 / 51.9	64.0 / 49.3	59.7 / 40.2	64.0 / 46.3	69.3 / 48.0	64.1 / 40.6	67.4 / 49.2
XLM-RLARGE	80.6 / 67.8	74.1 / 56.0	68.5 / 53.6	63.1 / 43.5	69.2 / 51.6	71.3 / 50.9	68.0 / 45.4	70.7 / 52.7
XLM-RLARGE (reimpl)	84.0 / 71.1	74.4 / 56.4	70.2 / 55.0	66.5 / 46.3	71.1 / 53.2	74.4 / 53.5	68.6 / 44.6	72.7 / 54.3
INFOXLMLARGE	84.5 / 71.6	75.1 / 57.3	71.2 / 56.2	67.6 / 47.6	72.5 / 54.2	75.2 / 54.1	69.2 / 45.4	73.6 / 55.2
Table 4: Evaluation results on MLQA cross-lingual question answering. We report the F1 and exact match (EM) scores. Results with “*” are taken from (Lewis et al., 2020). “(reimpl)” is our reimplementation of fine-tuning, which is the same as INFOXLM. Results of INFOXLM and XLM-R (reimpl) are averaged over five runs. “—XLCO” is the model without cross-lingual contrast.
Models	MLQA	XNLI	Average
MBERT	23.3	16.9	20.1
XLM-R	17.6	10.4	14.0
INFOXLM	15.8	10.3	13.1
-XLCo	16.1	11.0	13.6
contrast, INFOXLM still provides high retrieval accuracy at the last few layers, which indicates that INFOXLM provides better aligned representations than XLM-R. Moreover, we find that the performance is further improved when removing TLM, demonstrating that XLCo is more effective than TLM for aligning cross-lingual representations, although TLM helps to improve zero-shot cross-lingual transfer.
Effect of Cross-Lingual Pre-training Tasks To better understand the effect of the cross-lingual pre-training tasks, we perform ablation studies on the pre-training tasks of INFOXLM, by removing XLCo, TLM, or both. We present the experimental results in Table 7. Comparing the results of -TLM and -XLCo with the results of -TLM-XLCo, we find that both XLCo and TLM effectively improve cross-lingual transferability of the pre-trained INFOXLM model. TLM is more effective for XNLI while XLCo is more effective for MLQA. Moreover, the performance can be further improved by jointly learning XLCo and TLM.
Effect of Contrast on Universal Layer We conduct experiments to investigate whether contrast
Table 5: Cross-lingual transfer gap scores, i.e., averaged performance drop between English and other languages in zero-shot transfer. Smaller gap indicates better transferability. “-XLCo” is the model without cross-lingual contrast.
Model	XLCo Layer	XNLI	MLQA
INFOXLM15	8	76.45	67.87 / 49.58
INFOXLM15	12	76.12	67.83 / 49.50
INFOXLM15–TLM	8	75.58	67.42 / 49.27
INFOXLM15–TLM	12	75.85	67.84 / 49.54
Table 6: Contrast on the universal layer v.s. on the last layer. Results are averaged over five runs. “-TLM” is the ablation variant without TLM.
Model	XNLI	MLQA
[0]	INFOXLM15	76.45	67.87 / 49.58
[1]	[0]-XLCo	76.24	67.43 / 49.23
[2]	[0]-TLM	75.85	67.84 / 49.54
[3]	[2]-XLCo	75.33	66.86 / 48.82
[4]	[2]-Mixup	75.43	67.21 / 49.19
[5]	[2]-Momentum	75.32	66.58 / 48.66
Table 7: Ablation results on components of INFOXLM. Results are averaged over five runs.
on the universal layer improves cross-lingual pretraining. As shown in Table 6, we compare the evaluation results of four variants of INFOXLM, where XLCo is applied on the layer 8 (i.e., universal layer) or on the layer 12 (i.e., the last layer). We find that contrast on the layer 8 provides better results for INFOXLM. However, conducting XLCo on layer 12 performs better when the TLM task is excluded. The results show that maximizing context-sequence (TLM) and sequence-level (XLCo) mutual information at the last layer tends to interfere with each other. Thus, we suggest applying XLCo on the universal layer for pre-training INFOXLM.
Effect of Mixup Contrast We conduct an ablation study on the mixup contrast strategy. We pre
train a model that directly uses translation pairs for XLCo without mixup contrast  $(-TLM - Mixup)$ . As shown in Table 7, we present the evaluation results on XNLI and MLQA. We observe that mixup contrast improves the performance of INFOXLM on both datasets.
Effect of Momentum Contrast In order to show whether our pre-trained model benefits from momentum contrast, we pretrain a revised version of INFOXLM without momentum contrast. In other words, the parameters of the key encoder are always the same as the query encoder. As shown in Table 7, we report evaluation results (indicated by “-TLM-Momentum”) of removing momentum contrast on XNLI and MLQA. We observe a performance descent after removing the momentum contrast from INFOXLM, which indicates that momentum contrast improves the learned language representations of INFOXLM.
In this paper, we present a cross-lingual pre-trained model INFOXLM that is trained with both monolingual and parallel corpora. The model is motivated by the unified view of cross-lingual pretraining from an information-theoretic perspective. Specifically, in addition to the masked language modeling and translation language modeling tasks, INFOXLM is jointly pre-trained with a newly introduced cross-lingual contrastive learning task. The cross-lingual contrast leverages bilingual pairs as the two views of the same meaning, and encourages their encoded representations to be more similar than the negative examples. Experimental results on several cross-lingual language understanding tasks show that INFOXLM can considerably improve the performance.
Currently, most NLP research works and applications are English-centric, which makes non-English users hard to access to NLP-related services. Our work focuses on cross-lingual language model pretraining. With the pre-trained model, we are able to transfer end-task knowledge from high-resource languages to low-resource languages, which helps to build more accessible NLP applications. Additionally, incorporating parallel corpora into the pretraining procedure improves the training efficiency, which potentially reduces the computational cost for building multilingual NLP applications.
We appreciate the helpful discussions with Bo Zheng, Shaohan Huang, Shuming Ma, and Yue Cao. H.H. is the corresponding author. X.M. and H.H. are partly supported by National Key R&D Plan (No. 2018YFB1005100), National Natural Science Foundation of China (No. 61751201, 61602197 and 61772076), Natural Science Fund of Beijing (No. Z181100008918002), and the funds of Beijing Advanced Innovation Center for Language Resources (No. TYZ19005).
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. 2019. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 4623-4637. Association for Computational Linguistics.
Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7(0):597-610.
Philip Bachman, R Devon Hjelm, and William Buchwalter. 2019. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pages 15509-15519.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR.
Zewen Chi, Li Dong, Furu Wei, Wenhui Wang, XianLing Mao, and Heyan Huang. 2020. Cross-lingual natural language generation via pre-training. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7570-7577. AAAI Press.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020a. Unsupervised cross-lingual representation learning at scale. In
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In Advances in Neural Information Processing Systems, pages 7057-7067. Curran Associates, Inc.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b. Emerging cross-lingual structure in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6022–6034, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Philipp Dufter and Hinrich Schütze. 2020. Identifying elements essential for BERT's multilinguality. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4423-4437, Online. Association for Computational Linguistics.
Robert M. Fano. 1963. Transmission of Information: A Statistical Theory of Communications. M.I.T. Press.
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic BERT sentence embedding. arXiv preprint arXiv:2007.01852.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Olivier Henaff. 2020. Data-efficient image recognition with contrastive predictive coding. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4182-4192. PMLR.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411-4421. PMLR.
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. Unicoder: A universal language encoder by pretraining with multiple cross-lingual tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2485–2494, Hong Kong, China. Association for Computational Linguistics.
Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual bert: An empirical study. In International Conference on Learning Representations.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, San Diego, CA.
Lingpeng Kong, Cyprien de Masson d'Autume, Lei Yu, Wang Ling, Zihang Dai, and Dani Yogatama. 2020. A mutual information maximization perspective of language representation learning. In International Conference on Learning Representations.
Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. 2018. The IIT Bombay English-Hindi parallel corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, Miyazaki, Japan. European Language Resources Association.
Patrick Lewis, Barlas Oguz, Rudy Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315-7330, Online. Association for Computational Linguistics.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020. XGLUE: A new benchmark dataset for cross-lingual pre-training,