# A Survey on Machine Reading Comprehension--Tasks, Evaluation Metrics and Benchmark Datasets
### Outline  
In Section 2, we focus on the MRC tasks. We first give a definition of typical MRC task. Then we compare multi-modal MRCs with textual MRCs, and discuss the differences between question answering tasks and machine reading comprehension tasks. Next, we analyze the existing classification method of MRC tasks which is widely used by the community. We argue that the existing classification method is inadequate and has potential problems. In order to solve the above problems, we propose a more adequate classification method of MRC tasks. We summarize 4 different attributes of MRC tasks. Each of these attributes can be divided into several categories. We give a detailed definition of each category with examples and explain why the new classification method is more adequate. After that, we collect totally 57 different MRC tasks and categorize them according to the new classification method. Finally, we analyze these MRC tasks and make statistical tables and charts of them. In Section 3, we discuss the MRC evaluation metrics. Nine evaluation metrics of MRC tasks have been analyzed. We begin by presenting an overview of MRC evaluation metrics. Then we discuss the computing methods of each evaluation metric, including several sub-metrics such as token-level F1 and question-level F1. Next, we analyze the usage of each evaluation metric in different MRC tasks. After that, we make statistics on the usages of different evaluation metrics in the 57 MRC tasks. Finally, we analyze the relationship between the MRC task types and the evaluation metrics they used. In Section 4, we present the family of MRC datasets. We begin by analyzing the size of each MRC datasets. Here, we have counted the total number of questions in each MRC dataset along with the sizes of its training set, development set and testing set, as well as the proportion of training set. Then we discuss the generation method of datasets which can be roughly described as several categories: Crowdsourcing, Expert, and Automated. Next, we conduct an in-depth analysis of the source of corpus and the type of context of MRC datasets. After that, we try to find all the download links, leaderboards and baseline projects of MRC datasets, all of which have been published on our website. Then, we present a statistical analysis of prerequisite skills and citations of the papers in which each dataset was proposed. Next, we summarize 10 characteristics of MRC datasets. Finally, we give a detailed description of each MRC dataset. Figure 3: The number of MRC datasets created in recent years and the F1 scores of state-of-the-art models on SQuAD 1.1 [19]: (**a**) The cumulative number of MRC datasets from the beginning of 2014 to the end of 2019. (**b**) The progress of state-of-the-art models on SQuAD 1.1 since this dataset was released. The data points are taken from the leaderboard at [[https://rajpurkar.github.io/SQuAD-explorer/](https://rajpurkar.github.io/SQuAD-explorer/)]([https://rajpurkar.github.io/SQuAD-explorer/](https://rajpurkar.github.io/SQuAD-explorer/)). In Section 5, we discuss several open issues that remain unsolved in this field. Firstly, We believe that many important aspects have been overlooked which merit additional research, such as multi-modal MRC, commonsense and world knowledge, complex reasoning, robustness, interpretability, evaluation of the quality of MRC datasets. Secondly, we talk about understanding from the perspective of cognitive neuroscience. Finally, we share some of the latest research results of cognitive neuroscience and the inspiration of these results for NLP research. In Section 6, we present a comprehensive conclusion of this survey. Finally, we have published all the data on the website, the researchers could directly access each MRC datasets, papers, and baseline projects, or browse the leaderboards by clicking the hyperlinks. It is hoped that the research community could quickly access the comprehensive information of MRC datasets and their tasks.