SlowGuess/ACL_2020_OCR huggingface dataset:
We conduct ablative experiments to analyze the two modules of MMT, i.e., multi-source meta learning (MML) and meta transfer learning (MTL). The MTL is the transfer learning module specifically designed for MML, and TL denotes the traditional transfer learning without MML. The experiments are based on BERT-Base model, and all the results are reported in Table 5.
Table 4: Unsupervised domain adaptation on SWAG, where "M" denotes MCTEST, "R" denotes RACE, "DA" denotes decomposable attention, and "GLV" denotes GloVe vectors.
Dream	Dev	Test
BERT-Base	60.05	61.58
+MML(M)	49.85	52.87
+MML(R)	49.56	51.69
+MML(M∪R)	29.60	29.20
+TL(M)	60.31	60.14
+TL(R)	68.72	67.72
+TL(R-M)	68.97	67.38
+TL(M+R)	68.61	68.15
+MMT(M)	67.99	68.54
+MMT(R)	68.04	68.69
+MMT(M∪R)	61.72	60.12
MMT(M+R)	68.38	68.89
Table 5: Ablation study of MMT on DREAM. "TL" denotes supervised transfer learning, "M" denotes MCTest, "R" denotes RACE, and "U" denotes the task combination of RACE and MCTest.
In the first experiments, we present the results of the MML module. When the input source for MML is a single source, MML downgrades to the traditional meta learning. From the results, we observe that MML fine-tuned on MCTEST (MML(M)) is better than that on RACE (MML(R)), which is caused by the large difference between the RACE and DREAM datasets. We also compare the baseline that simply combines RACE and MCTEST datasets to be one large training source, denoted by MML(MUR), dramatically drops the performance and only achieves  $29.20\%$  on DREAM dataset, which is  $23.67\%$  lower than that of MML(M). This suggests that a simple combination of the two different training datasets for meta training is not a good choice.
For the transfer learning (TL) module, we can observe that the performance improvement is more significant by transferring knowledge from RACE to DREAM, compared to that from MCTest. In addition, TL(R-M) also benefits from fine-tuning on RACE and MCTest sequentially, and achieves better results.
With the help of MTL, MMT further boosts the performance on DREAM and outperforms both MML and TL baselines. For instance, MMT(M) outperforms MML(M) and TL(M) with  $15.67\%$  and  $8.40\%$ , respectively. Moreover, MMT is also helpful in alleviating the overfitting issue that exists in TL baselines. The results of development set for  $\mathrm{TL}(*)$  are higher than the test set, which indicates the poor generalization ability of  $\mathrm{TL}(*)$ . Fortunately, MMT(\*) is able to address this issue. The MMT(R+M) that is trained on both RACE and MCTEST in meta learning manner, achieves the best results in all evaluated methods.
5.2 Source Selection for MMT
Source selection is a prerequisite step for MMT. In previous experiments, we assume that training resources are given without selection. Due to the data heterogeneity of different sources, the performance of meta learning may drop if we incorporate some undesirable data sources in training. In this experiment, we evaluate the transferability between different datasets and further give the suggestion on the source selection for MMT. The results are summarized in Figure 4. In Figure, the X-axis denotes the source, and Y-axis denotes the target. The values in the boxes indicate transferability from source to the target data in terms of accuracy. For example, 14 denotes transferring RACE to the target MCTest will obtain  $14\%$  accuracy improvement over that only trained on the MCTest. The negative value in the transferability matrix suggests the negative transfer. There is no source that can be used to improve the performance of SWAG effectively.
Figure 4: Transferability matrix. X-axis denotes the source, and Y-axis denotes the target. The values indicate the transferability from source to the target data in terms of accuracy. The higher the value is, the stronger the transferability is. Taking MCTEST dataset for example, transfer learning pre-trained on RACE leads  $14\%$  performance improvement than fine-tuning on MCTEST only.
In MMT, we employ this transferability matrix to guide the source selection for MML training. Specifically, in supervised MMT, we only choose those training sources with the significant positive transfer. In unsupervised MMT, the source with the highest score is selected to be the initial state.
Figure 5: Training with different sources and test on SemEval. Where "S" denotes SWAG, "R" denotes RACE, "M" denotes MCTEST, and "D" denotes DREAM.
To verify the impact of different dataset to MMT, we further study the improvement on target SemEval by training with different sources. The results is shown in Figure 5. The performance of SemEval drops when we incorporate DREAM and SWAG into training. Recall the transferability matrix in Figure 4, the DREAM and SWAG datasets show little help in improving the performance on SemEval compared to RACE and MCTEST. In summary, more source data do not guarantee better performance. Only the "similar" source data will be beneficial for multi-source meta learning.
In this work, we propose a novel method named multi-source meta transfer for multiple-choice question answering on low resource setting. Our method considers multiple sources meta learning and target fine-tuning into a unified framework, which is able to learn a general representation from multiple sources and alleviate the discrepancy between source and target. We demonstrate the superiority of our methods on both supervised setting and unsupervised domain adaptation settings over the state-of-the-arts. In future work, we explore to extend this approach for other low resource tasks in NLP.
The paper is supported by the Agency for Science, Technology and Research (A*STAR) under its AME Programmatic Funding Scheme (Project No. A18A1b0045).
7338
Junwei Bao, Nan Duan, Zhao Yan, Ming Zhou, and Tiejun Zhao. 2016. Constraint-based question answering with knowledge graph. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2503-2514, Osaka, Japan. The COLING 2016 Organizing Committee.
Akshay Chaturvedi, Onkar Pandit, and Utpal Garain. 2018. CNN for text-based multiple choice question answering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 272-277, Melbourne, Australia. Association for Computational Linguistics.
Jifan Chen and Greg Durrett. 2019. Understanding dataset design choices for multi-hop reasoning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4026-4032, Minneapolis, Minnesota. Association for Computational Linguistics.
Zhipeng Chen, Yiming Cui, Wentao Ma, Shijin Wang, Ting Liu, and Guoping Hu. 2018. Hfl-rc system at semeval-2018 task 11: hybrid multi-aspects model for commonsense reading comprehension. arXiv preprint arXiv:1803.05655.
Yu-An Chung, Hung-Yi Lee, and James Glass. 2018. Supervised and unsupervised transfer learning for question answering. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1585-1594, New Orleans, Louisiana. Association for Computational Linguistics.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence, Italy. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378, Minneapolis, Minnesota. Association for Computational Linguistics.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. 2019. On the convergence theory of gradient-based model-agnostic meta-learning algorithms. arXiv preprint arXiv:1908.10400.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, pages 1126-1135, Sydney, NSW, Australia. JMLR.org.
Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li, and Kyunghyun Cho. 2018. Meta-learning for low-resource neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3622-3631, Brussels, Belgium. Association for Computational Linguistics.
Daya Guo, Duyu Tang, Nan Duan, Ming Zhou, and Jian Yin. 2019. Coupling retrieval and meta-learning for context-dependent semantic parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 855-866, Florence, Italy. Association for Computational Linguistics.
Shangmin Guo, Kang Liu, Shizhu He, Cao Liu, Jun Zhao, and Zhuoyu Wei. 2017. IJCNLP-2017 task 5: Multi-choice question answering in examinations. In Proceedings of the IJCNLP 2017, Shared Tasks, pages 34-40, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Po-Sen Huang, Chenglong Wang, Rishabh Singh, Wentau Yih, and Xiaodong He. 2018. Natural language to structured query generation via meta-learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 732-738, New Orleans, Louisiana. Association for Computational Linguistics.
Di Jin, Shuyang Gao, Jiun-Yu Kao, Tagyoung Chung, and Dilek Hakkani-tur. 2019. Mmm: Multi-stage multi-task learning for multi-choice reading comprehension. arXiv preprint arXiv:1910.00458.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 252-262, New Orleans, Louisiana. Association for Computational Linguistics.
7339
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1378-1387, New York, New York, USA. PMLR.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAd-ing comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794, Copenhagen, Denmark. Association for Computational Linguistics.
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. 2011. One shot learning of simple visual concepts. In Proceedings of the annual meeting of the cognitive science society, volume 33, pages 2573-2575, Boston, Massachusetts, USA. Curran Associates, Inc.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xi-ang Ren. 2019. KagNet: Knowledge-aware graph networks for commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2829-2839, Hong Kong, China. Association for Computational Linguistics.
Xiaodong Liu, Yelong Shen, Kevin Duh, and Jianfeng Gao. 2018. Stochastic answer networks for machine reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1694-1704, Melbourne, Australia. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. ArXiv, abs/1907.11692.
Robert Munro, Daren Ler, and Jon Patrick. 2003. Meta-learning orthographic and contextual models for language independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 192-195, Edmonton, Canada. Association for Computational Linguistics.
Alex Nichol, Joshua Achiam, and John Schulman. 2018. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999.
Abiola Obamuyide and Andreas Vlachos. 2019. Model-agnostic meta-learning for relation classification with limited supervision. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5873-5879, Florence, Italy. Association for Computational Linguistics.
Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. 2018. SemEval-2018 task 11: Machine comprehension using commonsense knowledge. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 747-757, New Orleans, Louisiana. Association for Computational Linguistics.
Alessio Palmero Aprosio, Sara Tonelli, Marco Turchi, Matteo Negri, and Mattia A. Di Gangi. 2019. Neural text simplification in low-resource conditions using weak supervision. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 37-44, Minneapolis, Minnesota. Association for Computational Linguistics.
Kun Qian and Zhou Yu. 2019. Domain adaptive dialog generation via meta learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2639-2649, Florence, Italy. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 193-203, Seattle, Washington, USA. Association for Computational Linguistics.
Rico Sennrich and Biao Zhang. 2019. Revisiting low-resource neural machine translation: A case study. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 211-221, Florence, Italy. Association for Computational Linguistics.
Hongsuck Seo, Jonghoon Lee, Seokhwan Kim, Kyusong Lee, Sechun Kang, and Gary Geunbae Lee. 2012. A meta learning approach to grammatical error correction. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 328-332,
7340
Jeju Island, Korea. Association for Computational Linguistics.
Shashank Srivastava, Igor Labutov, and Tom Mitchell. 2018. Zero-shot learning of classifiers from natural language quantification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 306-316, Melbourne, Australia. Association for Computational Linguistics.
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019a. DREAM: A challenge data set and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics, 7:217-231.
Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2019b. Improving machine reading comprehension with general reading strategies. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2633-2643, Minneapolis, Minnesota. Association for Computational Linguistics.
Shuohang Wang, Mo Yu, Jing Jiang, and Shiyu Chang. 2018. A co-matching model for multi-choice reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 746-751, Melbourne, Australia. Association for Computational Linguistics.
Jiawei Wu, Wenhan Xiong, and William Yang Wang. 2019. Learning to learn and predict: A meta-learning approach for multi-label classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4345-4355, Hong Kong, China. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32, pages 5754-5764, New York, USA. Curran Associates, Inc.
Jianxing Yu, Zhengjun Zha, and Jian Yin. 2019. Inferential machine comprehension: Answering questions by recursively deducing the evidence chain from text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2241-2251, Florence, Italy. Association for Computational Linguistics.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93-104, Brussels, Belgium. Association for Computational Linguistics.
Zhenjie Zhao and Xiaojuan Ma. 2019. Text emotion distribution learning from small sample: A meta-learning approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3948-3958, Hong Kong, China. Association for Computational Linguistics.
7341