SlowGuess/ACL_2020_OCR huggingface dataset:
In terms of LAKM task, there are decent gains for all languages, including English. However, the gains are bigger on low resource languages compared with English performance. Take XLM and MLQA dataset as an example, LAKM gets  $1.8\%$  and  $3.2\%$  EM improvements on es and de, while the improvement on en is about  $0.5\%$ . The intuition behind en gains is that LAKM brings extra data with knowledge to en as well.
In terms of mixMRC task, there are slight regression on en compared with decent gains on es, de and fr. Take XLM and MTQA dataset for illustrations, mixMRC has  $0.6\%$  EM regression on en versus  $1.4\%$  and  $0.5\%$  EM gains on fr and de languages. This shows that mixMRC mainly improves the transferring capability from rich resource language to low resource language.
Model	Methods	MLQA (EM / F1)	MTQA (EM / F1)
en	es	de	en	fr	de
M-BERT	Lewis et al. (2019)	65.2 / 77.7	37.4 / 53.9	47.5 / 62.0	-	-	-
Baseline	65.4 / 79.0	50.4 / 68.5	46.2 / 60.6	67.0 / 86.9	52.9 / 78.2	59.8 / 81.4
LAKM	66.9 / 80.1	51.5 / 69.5	49.9 / 64.4	68.8 / 87.6	56.8 / 78.8	62.4 / 81.9
mixMRC	65.4 / 79.4	50.5 / 69.1	49.1 / 64.0	67.9 / 86.8	56.4 / 77.8	62.4 / 81.0
mixMRC + LAKM	64.7 / 79.2	52.1 / 70.4	50.9 / 65.6	68.6 / 87.0	57.5 / 78.5	62.9 / 81.3
XLM	Lewis et al. (2019)	62.4 / 74.9	47.8 / 65.2	46.7 / 61.4	-	-	-
Baseline	64.1 / 77.6	50.4 / 68.4	47.4 / 62.0	67.1 / 86.8	51.5 / 75.8	61.6 / 81.3
LAKM	64.6 / 79.0	52.2 / 70.2	50.6 / 65.4	68.3 / 87.3	52.5 / 75.9	61.9 / 81.2
mixMRC	63.8 / 78.0	52.1 / 69.9	49.8 / 64.8	66.5 / 85.9	52.9 / 75.0	62.1 / 80.5
mixMRC + LAKM	64.4 / 79.1	52.2 / 70.3	51.2 / 66.0	68.2 / 86.8	53.6 / 75.9	62.5 / 80.9
In this section, we ablate important components in LAKM to explicitly demonstrate its effectiveness.
5.1 Random N-gram Masking vs LAKM
To study the effectiveness of LAKM, we compare LAKM with Random N-gram Masking $^{\dagger}$  based on XLM and MTQA dataset. LAKM and Random N-gram Masking refer to fine-tuning XLM with the language-specific knowledge masking strategy and random n-gram masking strategy respectively. As shown in Table 8, without the language-agnostic knowledge masking strategy, the EM metrics drops by  $0.2\% - 0.87\%$ , which proves the necessity of LAKM.
Table 7: Experimental results on MLQA and MTQA dataset under translation condition  $(\%)$
Setting (EM)	en	fr	de
Random N-gram Masking	67.5	51.8	61.7
LAKM	68.3	52.5	61.9
5.2 Zero Shot Fine-tuning w/ vs w/o LAKM
To illustrate the effectiveness of the auxiliary tasks, an extreme scenario is considered when only English training data is available and there is no translation data. That means that we are unable to use mixMRC task to driver more accurate answer span boundaries. At this point, we only leverage LAKM to enhance answer boundary detection and compares the performance of M-BERT baseline with our model in Table 9.
From the experimental results, zero shot finetuning with LAKM is significantly better than MBERT baseline. On MTQA, our model gets  $2\%$ ,
3.3%, 3.8% EM improvements on English, French and German respectively. On MLQA, we get 1.6%, 1.4%, 1.2% EM improvements on English, Spanish and German.
Table 8: Ablation study on MTQA  $(\%)$ .
MLQA (EM / F1)
en	es	de
Baseline	65.2 / 77.7	46.6 / 64.3	44.3 / 57.9
LAKM	66.8 / 80.0	48.0 / 65.9	45.5 / 60.5
MTQA (EM / F1)
en	fr	de
Baseline	65.8 / 86.6	41.3 / 70.9	50.7 / 76.2
LAKM	67.8 / 87.2	44.6 / 72.1	54.5 / 77.8
Table 9: Zero Shot experimental results on MLQA and MTQA datasets (\%). We only use English MRC training data and don't use translation data.
5.3 Extensive Analysis on Fine-grained Answer Types
To have an insight that how the new tasks (LAKM/mixMRC) affect the multilingual MRC task, we further analyze model performance on various answer types, as shown in Figure 5.
The comparison with baseline indicates that in most of the answer types (like color, description, money), both LAKM and mixMRC can enhance the answer boundary detection for multilingual MRC task.
One interesting finding is that in terms of animal, full_name, LAKM outperforms mixMRC by a great margin, which are  $9.1\%$  and  $14.3\%$  respectively. One possible explanation is that the knowledge phrases of LAKM can cover some entity related phrases like animals and names, leading to the significant EM boost.
In terms of those numerical answer types (like money, numeric, length), the per
$^{\dagger}$ Random N-gram Masking shows gains in English SQuAD.
formance between mixMRC and LAKM are similar. The intuition behind this is that these numerical answers may be easier to transfer between different languages since answers like length are similar across different languages.
Figure 5: EM results comparison on M-BERT (MTQA French test set) for the different answer types.
This paper proposes two auxiliary tasks (mixMRC and LAKM) in the multilingual MRC fine-tuning stage to enhance answer boundary detection especially for low resource languages. Extensive experiments on two multilingual MRC datasets have been conducted to prove the effective of our proposed approach. Meanwhile, we further analyze the model performance on fine-grained answer types, which shows interesting insights.
Akari Asai, Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2018. Multilingual extractive reading comprehension by runtime machine translation. arXiv preprint arXiv:1809.03275.
Giuseppe Castellucci, Valentina Bellomaria, Andrea Favalli, and Raniero Romagnoli. 2019. Multilingual intent detection and slot filling in a joint bert-based model. arXiv preprint arXiv:1907.02884.
Xilun Chen, Ahmed Hassan Awadallah, Hany Hassan, Wei Wang, and Claire Cardie. 2019. Multi-source cross-lingual model transfer: Learning what to share. In Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL).
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In Advances in Neural Information Processing Systems, pages 7057-7067.
Alexis Conneau, Guillaume Lample, Rudy Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating crosslingual sentence representations. arXiv preprint arXiv:1809.05053.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2019. Cross-lingual machine reading comprehension. arXiv preprint arXiv:1909.00361.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693-1701.
Minghao Hu, Yuxing Peng, Zhen Huang, Xipeng Qiu, Furu Wei, and Ming Zhou. 2017. Reinforced mnemonic reader for machine reading comprehension. arXiv preprint arXiv:1705.02798.
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. Unicoder: A universal language encoder by pretraining with multiple cross-lingual tasks. arXiv preprint arXiv:1909.00964.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv:1707.07328.
Yimin Jing, Deyi Xiong, and Yan Zhen. 2019. Bipar: A bilingual parallel dataset for multilingual and cross-lingual reading comprehension on novels. arXiv preprint arXiv:1910.05040.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551.
Phillip Keung, Yichao Lu, and Vikas Bhardwaj. 2019. Adversarial learning with contextual embeddings for zero-resource cross-lingual classification and ner. arXiv preprint arXiv:1909.00153.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. arXiv preprint arXiv:1901.07291.
Kyungjae Lee, Kyoungho Yoon, Sunghyun Park, and Seung-won Hwang. 2018. Semi-supervised Training Data Generation for Multilingual Question Answering. page 5.
Patrick Lewis, Barlas Oguz, Rudy Rinott, Sebastian Riedel, and Holger Schwenk. 2019. *Mlqa: Evaluating cross-lingual extractive question answering*. arXiv preprint arXiv:1910.07475.
Todor Mihaylov and Anette Frank. 2018. Knowledgeable reader: Enhancing cloze-style reading comprehension with external commonsense knowledge. arXiv preprint arXiv:1805.07858.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? arXiv preprint arXiv:1906.01502.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad:  $100,000+$  questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hananneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In International Conference on Learning Representations.
Jasdeep Singh, Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2019. Xlda: Cross-lingual data augmentation for natural language inference and question answering. arXiv preprint arXiv:1905.11471.
Yibo Sun, Daya Guo, Duyu Tang, Nan Duan, Zhao Yan, Xiaocheng Feng, and Bing Qin. 2018. Knowledge based machine reading comprehension. arXiv preprint arXiv:1809.04267.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2019. Ernie 2.0: A continual pre-training framework for language understanding.
Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. 2019. Cross-lingual bert transformation for zero-shot dependency parsing. arXiv preprint arXiv:1909.06775.
Zhiguo Wang, Haitao Mi, Wael Hamza, and Radu Florian. 2017. Multi-perspective context matching for machine comprehension. arXiv preprint arXiv:1612.04211.
Dirk Weissenborn, Tomáš Kočisky, and Chris Dyer. 2017. Dynamic integration of background knowledge in neural nlu systems. arXiv preprint arXiv:1706.02596.
Caiming Xiong, Victor Zhong, and Richard Socher. 2018. Dynamic coattention network for question answering. US Patent App. 15/421,193.
Bishan Yang and Tom Mitchell. 2017. Leveraging knowledge bases in LSTMs for improving machine reading. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1436-1446, Vancouver, Canada. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. Ernie: Enhanced language representation with informative entities. arXiv preprint arXiv:1905.07129.