SlowGuess/NAACL_2022_OCR huggingface dataset:
Pre-training Details In our implementation of MLM, we follow the setting in (Devlin et al., 2018), which randomly mask  $15\%$  tokens of the input sequence<sup>4</sup>. We optimize our model with the Adam optimizer, using batch size of 64 for a total of 15K steps in pre-training. And learning rate is set to 1e-5 with 1.5K warmup steps. The total max input sequence length is set to 256, that is, the max length of each sentence is set to 128. Notice that  $\tau$  in Eq.7 is set to 20. We pre-train our model using  $8\times \mathrm{V}100 - 32\mathrm{G}$  GPUs for 4-5 hours. Fine-tuning details can be seen in the Appendix C.
We conduct experiments over two xSL tasks: xMRC and xNER including four datasets: MLQA (Lewis et al., 2020), XQUAD (Artetxe et al., 2019b), CoNLL (Sang, 2002) and WikiAnn (Pan et al., 2017). Detailed introduction about these datasets presented in the Appendix D.
For all datasets, we fine-tune our model in zero-shot setting, which denotes we only utilize the English training set to optimize the model, and test the resulting model on other target languages.
For xMRC task, we use two evaluation metrics, Exact Match (EM) and span-level F1 score, which are popularly used for accuracy evaluation of MRC models. Span-level F1 measures the span overlap between ground-truth answer and model predictions. Exact match (EM) score is 1 if the prediction is exactly the same as the ground truth, otherwise 0. As for xNER task, we use entity-level F1 score to evaluate our model, which requires the boundary and type between the prediction and the ground-truth entity should be matched precisely.
Baselines We compare our model with the following pre-trained baselines: (1) M-BERT (Devlin et al., 2018) pre-trained with MLM task on Wikipedia data in 102 languages; (2) XLM (Conneau and Lample, 2019), another multilingual PLM pre-trained with MLM and TLM tasks in 100 languages; (3) XLM-R (Conneau et al., 2019), a strong effective xPLM pre-trained with MLM in more large corpus; (4) Info-XLM (Chi et al., 2021a),
4The pre-defined answers in CLISM will not be masked to avoid missing labels.
Models	#data	#languages	MLQA	XQUAD	CoNLL	WikiAnn
M-BERT	-	102	57.80/42.40	69.63/53.72	78.2	69.1
XLM	-	100	61.70/44.20	70.93/53.18	79	69.5
CalibreNet	-	-	65.40/48.84	-	80.92	-
Chen et al. (2021b)	-	-	66.00/48.88	75.06/59.87	-	-
XLM-R	-	-	64.14/46.00	73.54/57.55	78.48	70.28
Info-XLM*	130M	94	65.85/48.23	75.79/59.50	79.52	72.23
XLM-Align*	130M	94	65.66/48.47	75.35/59.10	78.87	72.66
Ours (XLM-R)*	1M	4	67.34/49.11	76.73/60.87	80.63	73.31
Ours (Info-XLM)	1M	4	69.07/50.81	78.22/61.99	81.13	73.87
Table 2: Average evaluation results on four datasets. #data denotes the amount of pre-training data. #languages represents the language diversities in pre-training data. The results of our model are averaged over 5 runs. * denotes the model build upon of XLM-R. The results of each language are represented in the Appendix E.
a recent xPLM which continues training XLM-R with MLM, TLM and XLCO in 94 languages; (5) XLM-Align (Chi et al., 2021b), a more recent but effective xPLM, also built on top of XLM-R with MLM and DWA in 94 languages. Besides, we also utilize strong baselines which are specifically designed for xSL tasks: (1) CalibreNet (Liang et al., 2021b) designed for enhancing the boundary detection of xPLMs in xSL tasks; (2) AA-CL (Chen et al., 2021b), a more recent approach to construct hard-negatives via contrastive learning for xMRC. Note that, we reimplement the base version of these baselines in our local environment with 3 times and report the average results of baselines.
We represent two main results in full-data setting and few-shot data setting. That is, our model utilizes all data in training set to train the model in full-data setting. In contrast, the model is finetuned with smaller training sets in few-shot data setting.
xMRC Results Table 2 shows the comparison between our approach and representative systems on four datasets. Specifically, our models surpass these baselines by a large margin on two xMRC datasets. For examples, our model initialized from Info-XLM achieve  $69.07\% /50.81\%$  (vs.  $65.85\% /48.23\%$ ) in terms of F1/EM score. Compared with Info-XLM or XLM-Align, which both build on top of XLM-R with 130 million data across 94 languages, our model initialized from XLM-R obtains better performances like  $67.34\% /48.91\%$  (vs.  $65.85\% /48.23\%$ ) on MLQA dataset. It is worthwhile to indicate that our model only pre-tained with 1 million parallel data across
4 languages. The results bear out the effectiveness of our proposed methods even if with limited pre-training data across several languages (1M≪130M).
Also when looked into the Table 3, our models dramatically improve performances in the more challenging few-shot data setting. Given a small training set with only 512 examples, our model built on top of XLM-R improve the baseline by  $24\% / 20.4\%$  and  $20.59\% / 18\%$  in terms of F1/EM score on both two datasets. The results show that our method presents a new paradigm for xSL tasks in few-shot data setting.
xNER Results As presented in Table 2, first compared with baseline XLM-R, our resulting model on top of XLM-R achieves an average F1 score gain of  $1.15\%$  and  $3.03\%$ . Then compared with Info-XLM and XLM-Align on top of XLM-R which pre-trained on 130 million large corpus across 94 languages, ours still show superior performances. That is, with less than one percent of the corpus they used, our approach has shown its full potential.
In this section, we first discuss the importance of each key component in our model. Then we conduct an analysis of alleviating pretrain-finetune discrepancy by ours and independent contribution of the proposed CLISM at few-shot data setting. We further analyse the relation between pre-training data and our methods in the Appendix G.
Key components We perform an ablation study to better understand the independent contributions
(b)
Figure 4: (a): Average F1 score of both Ours and base architecture XLM-R on the XQUAD test set at different training steps. (b): Average F1 score of all methods with giving different few-shot training examples on the XQUAD test set. It is worthwhile to notice that ours denotes the resulting model build on top of XLM-R in these experiments.
Models	MLQA	XQUAD
64 examples
XLM-R	6/1.5	8/2
Info-XLM	7/3.2	8.7/5.1
Ours (XLM-R)	14/7.2	15.5/8.2
128 examples
XLM-R	10/3.5	12/4
Info-XLM	11.4/4.5	15.7/7
Ours (XLM-R)	26.4/15.9	29.5/19.7
512 examples
XLM-R	26.8/15.5	33.35/19.1
Info-XLM	29.7/18.9	36.0/21.0
Ours (XLM-R)	50.8/35.9	53.94/37.1
Table 3: Model performances across xMRC datasets when the number of training examples is 64, 128 and 512. The more comprehensive results in few-shot data setting are described in Appendix F.
Algorithms	MLQA	XQUAD
Ours (XLM-R)	67.34/49.11	76.73/60.87
w/o CLISM	64.56/46.80	74.34/58.43
w/o CACR	66.40/48.01	75.87/59.95
w/o MLM	66.18/47.89	75.34/59.61
Table 4: Ablation study of pre-training schemes.
of the proposed pre-training schemes. Table 4 shows the performances when removing each key component of our model on MLQA and XQUAD datasets. Obviously, when removing CLISM, the model performances drop dramatically in both two datasets, showing the effectiveness of the proposed pre-training task. Meanwhile, we observe that with CACR or MLM also giving performance improvement, proving each of them plays an important role in our pre-training scheme.
Analysis of alleviating pretrain-finetune discrepancy We conduct extra experiments to verify the
proposed methods whether alleviate the pretrainfinetune discrepancy in cross-lingual sequence labeling tasks. As shown in the Figure 4 (a), in the early stage of training, ours manages to maintain an absolute lead of  $5\% -15\%$  on F1 score over XLM-R consistently, showing the resulting model not only speed the fine-tuning time but also achieve superior performances. This demonstrates representations learned by ours are more suitable for data distribution in xSL tasks, leading to better performance.
Independent Contribution of CLISM Moreover, in order to understand the independent contribution of the proposed CLISM at few-shot data setting. Figure 4 (b) gives the performance of individual models when giving  $\{64,128,256,512,1024\}$  examples. It is evident that our model outperforms baseline by large margins. Interestingly, we conduct the following experiments to verify effectiveness of using the [QUE] token during fine-tuning: We don't append [QUE] in the input sequence when fine-tuning our model in down-stream tasks, and thus, we can not utilize its representation as question. Thereafter, we follow the standard setting (Liang et al., 2021a; Chen et al., 2021b) to fine-tune ours in xSL tasks. Nevertheless, as shown in the 4 (b), the model performance drops heavily especially at smaller training data setting, such as 64, 128 examples. We interpret this phenomenon as follows: After pre-training, the [QUE] token captures much information about question that can be used to select the span from context in xSL tasks.
In this paper, heading for bridging the discrepancy including the objective and representation gap between pre-training and fine-tuning stages in cross
lingual sequence labeling tasks. We propose two pre-training strategies: cross-lingual language informative span masking (CLISM) and ContrAstive-Consistency Regularization (CACR), reaching surprisingly superior results on various benchmarks even when only a hundred examples available. We further prove that even with less than a hundredth of the data needed to train other xPLMs, our model still achieves better performances. As an extension of our future work, we will explore how to apply our methods to other nlp tasks.
Jian Pei's research is supported in part by the NSERC Discovery Grant program. All opinions, findings, conclusions and recommendations in this paper are those of the authors and do not necessarily reflect the views of the funding agencies.
Houssam Akhmouch, Gael Dias, and Jose G. Moreno. 2021. Understanding feature focus in multitask settings for lexico-semantic relation identification. In ACL/IJCNLP (Findings), volume ACL/IJCNLP 2021 of Findings of ACL, pages 2762-2772. Association for Computational Linguistics.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019a. On the cross-lingual transferability of monolingual representations. CoRR, abs/1910.11856.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019b. On the cross-lingual transferability of monolingual representations. CoRR, abs/1910.11856.
Akari Asai, Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2018. Multilingual extractive reading comprehension by runtime machine translation. CoRR, abs/1809.03275.
Nuo Chen, Fenglin Liu, Chenyu You, Peilin Zhou, and Yuexian Zou. 2021a. Adaptive bi-directional attention: Exploring multi-granularity representations for machine reading comprehension. In ICASSP, pages 7833-7837. IEEE.
Nuo Chen, Linjun Shou, Min Gong, Jian Pei, and Daxin Jiang. 2021b. From good to best: Two-stage training for cross-lingual machine reading comprehension.
Nuo Chen, Chenyu You, and Yuexian Zou. 2021c. Self-supervised dialogue learning for spoken conversational question answering. arXiv preprint arXiv:2106.02182.
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021a. Infoxm: An information-theoretic framework for cross-lingual
language model pre-training. In NAACL-HLT, pages 3576-3588. Association for Computational Linguistics.
Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, XianLing Mao, Heyan Huang, and Furu Wei. 2021b. Improving pretrained cross-lingual language models via self-labeled word alignment. In ACL/IJCNLP (1), pages 3418-3430. Association for Computational Linguistics.
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, and Furu Wei. 2021c. XLM-E: cross-lingual language model pre-training via ELECTRA. CoRR, abs/2106.16138.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116.
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In NeurIPS, pages 7057-7067.
Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. 2021. Template-based named entity recognition using BART. In ACL/IJCNLP (Findings), volume ACL/IJCNLP 2021 of Findings of ACL, pages 1835-1845. Association for Computational Linguistics.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2019. Cross-lingual machine reading comprehension. In EMNLP/IJCNLP (1), pages 1586–1595. Association for Computational Linguistics.
Xiang Deng, Yu Su, Alyssa Lees, You Wu, Cong Yu, and Huan Sun. 2021. Reasonbert: Pre-trained to reason with distant supervision. In EMNLP (1), pages 6112-6127. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. 2018. Dureader: a chinese machine reading comprehension dataset from real-world applications. In QA@ACL, pages 37-46. Association for Computational Linguistics.
Karl Moritz Hermann, Tomás Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS, pages 1693-1701.
R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep
representations by mutual information estimation and maximization. In ICLR. OpenReview.net.
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks. In EMNLP/IJCNLP (1), pages 2485-2494. Association for Computational Linguistics.
Fangkai Jiao, Yangyang Guo, Yilin Niu, Feng Ji, FengLin Li, and Liqiang Nie. 2021. REPT: bridging language models and machine reading comprehension via retrieval-based pre-training. In ACL/IJCNLP (Findings), pages 150-163. Association for Computational Linguistics.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In HLT-NAACL, pages 260–270. The Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In ICLR. Open-Review.net.
Patrick S. H. Lewis, Barlas Oguz, Rudy Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: evaluating cross-lingual extractive question answering. In ACL, pages 7315-7330. Association for Computational Linguistics.
Shining Liang, Ming Gong, Jian Pei, Linjun Shou, Wanli Zuo, Xianglin Zuo, and Daxin Jiang. 2021a. Reinforced iterative knowledge distillation for crosslingual named entity recognition. In KDD, pages 3231-3239. ACM.
Shining Liang, Linjun Shou, Jian Pei, Ming Gong, Wanli Zuo, and Daxin Jiang. 2021b. Calibrenet: Calibration networks for multilingual sequence labeling. In WSDM, pages 842-850. ACM.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020. XGLUE: A new benchmark dataset for cross-lingual pre-training, understanding and generation. In EMNLP (1), pages 6008-6018. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen. 2021. Text2event: Controllable sequence-to-structure generation for end-to-end event extraction. In ACL/IJCNLP (1), pages 2795-2806. Association for Computational Linguistics.
Fuli Luo, Pengcheng Yang, Shicheng Li, Xuancheng Ren, Xu Sun, Songfang Huang, and Fei Huang. 2021. Rethinking denoised auto-encoding in language pretraining. In EMNLP (1), pages 2922–2932. Association for Computational Linguistics.
Qing Lyu, Hongming Zhang, Elior Sulem, and Dan Roth. 2021. Zero-shot event extraction via transfer learning: Challenges and insights. In ACL/IJCNLP (2), pages 322-332. Association for Computational Linguistics.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In ACL (1), pages 1946–1958. Association for Computational Linguistics.
Shichao Pei, Lu Yu, Guoxian Yu, and Xiangliang Zhang. 2020. REA: robust cross-lingual entity alignment between knowledge graphs. In KDD, pages 2175-2184. ACM.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? arXiv preprint arXiv:1906.01502.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. In ACL (2), pages 784-789. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,  $000+$  questions for machine comprehension of text. In EMNLP, pages 2383-2392. The Association for Computational Linguistics.
Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, and Omer Levy. 2021. Few-shot question answering by pretraining span selection. In ACL/IJCNLP (1), pages 3066-3079. Association for Computational Linguistics.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. Coqa: A conversational question answering challenge. Trans. Assoc. Comput. Linguistics, 7:249-266.
Erik F. Tjong Kim Sang. 2002. Introduction to the conll-2002 shared task: Language-independent named entity recognition. In CoNLL. ACL.
Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In *ICLR (Poster)*. OpenReview.net.
Yongliang Shen, Xinyin Ma, Zeqi Tan, Shuai Zhang, Wen Wang, and Weiming Lu. 2021. Locate and label: A two-stage identifier for nested named entity recognition. In ACL/IJCNLP (1), pages 2782-2794. Association for Computational Linguistics.
Jasdeep Singh, Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2019. XLDA: cross-lingual data augmentation for natural language inference and question answering. CoRR, abs/1905.11471.
Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. ERNIE 2.0: A continual pre-training framework for language understanding. In AAAI, pages 8968-8975. AAAI Press.
Yuval Varkel and Amir Globerson. 2020. Pre-training mention representations in coreference models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8534-8540, Online. Association for Computational Linguistics.
Mingzhu Wu, Nafise Sadat Moosavi, Dan Roth, and Iryna Gurevych. 2021. Coreference reasoning in machine reading comprehension. In ACL/IJCNLP (1), pages 5768-5781. Association for Computational Linguistics.
Chenyu You, Nuo Chen, Fenglin Liu, Dongchao Yang, and Yuexian Zou. 2020a. Towards data distillation for end-to-end spoken conversational question answering. CoRR, abs/2010.08923.
Chenyu You, Nuo Chen, and Yuexian Zou. Mrd-net: Multi-modal residual knowledge distillation for spoken question answering.
Chenyu You, Nuo Chen, and Yuexian Zou. 2020b. Contextualized attention-based knowledge transfer for spoken conversational question answering. arXiv preprint arXiv:2010.11066.
Chenyu You, Nuo Chen, and Yuexian Zou. 2021a. Knowledge distillation for improved accuracy in spoken question answering. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7793-7797. IEEE.
Chenyu You, Nuo Chen, and Yuexian Zou. 2021b. Self-supervised contrastive cross-modality representation learning for spoken question answering. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 28-39.
Chenyu You, Ruihan Zhao, Lawrence Staib, and James S Duncan. 2021c. Momentum contrastive voxel-wise representation learning for semi-supervised volumetric medical image segmentation. arXiv preprint arXiv:2105.07059.
Chenyu You, Yuan Zhou, Ruihan Zhao, Lawrence Staib, and James S Duncan. 2022. Simcvd: Simple contrastive voxel-wise representation distillation for semi-supervised medical image segmentation. IEEE Transactions on Medical Imaging.
Fei Yuan, Linjun Shou, Xuanyu Bai, Ming Gong, Yaobo Liang, Nan Duan, Yan Fu, and Daxin Jiang. 2020. Enhancing answer boundary detection for multilingual machine reading comprehension. In ACL, pages 925-934. Association for Computational Linguistics.
The main contributions of this paper are towards tackling fundamental issue during pre-training and fine-tuning in xSL tasks when leveraging pretrained language models. The proposed method alleviates the discrepancy between pretrain-finetune paradigm, and thus improving the results of xPLMs in these tasks. Therefore, ideas of this paper can be applied in xPLM-based application systems such as cross-lingual question answering systems or robots. More generally, we expect the core idea of this paper can give insights of other research communities who want to build a more robust and effective xPLM-based model in their down-stream tasks. Admittedly, the proposed strategies are restricted with cross-lingual sequence labeling tasks, and biases in training dataset also influence the performance of the resulting model. These concerns warrant further research and consideration when utilizing this work to build cross-lingual application systems.
B Informative Span Selection
As illustrated in the paper Section 3.1, it is essential to select semantically meaningful spans before pre-training. And some extracted entities could be unreasonable due to the performance limit of off-the-shelf NER tools. Hence, we make several rules to filer out some recurring spans which tend to be uninformative:
- Spans which only consist of stop words will be filtered out.
- The boundary of spans must be the words.
- The max sequence length of each span is limited to 10.
C Fine-tuning Settings
Fine-tuning Details We train our model on two xSL tasks: xMRC and xNER. For both two tasks, we utilize Adam optimizer to train the model and batch size is set to 32. For xMRC, the learning rate is 3e-5, the maximum sequence length is set to 384. We train our model using  $8 \times \mathrm{V}100 - 32\mathrm{G}$  GPUs with 5 epochs for fine-tuning. For xNER, the learning rate is set to 5e-5, the max length is 128. It takes 5 epochs with using  $8 \times \mathrm{V}100 - 32\mathrm{G}$  GPUs to get the best checkpoint of our model.
Few-shot Data Details We also conduct few-shot data experiments on downstream datasets in xMRC. For each, dataset, we random sample few-shot training datasets from the original full-data sets, with the sampling size \{64, 128, 256, 512, 1024\}. To reduce variance, for each few-shot training dataset size, we random sample 5 times with different seeds, and report average results. When training our model on few-shot datasets, we choose 200 steps for optimizing the resulting model.
D xSL datasets
Cross-Lingual Machine Reading Comprehension MLQA and XQUAD are two popular xMRC benchmarks, which provide train dataset in English and test datasets in multiple low-resouce language. In this work, we evaluate our methods on six languages: including English, Arabic, German, Spanish, Hindi, Vietnamese.