# A Survey on Machine Reading Comprehension--Tasks, Evaluation Metrics and Benchmark Datasets
## 4 Benchmark Dataset
### The Generation Method of Datasets  
The generation method of datasets can be roughly described into several categories: Crowdsourcing, Expert, and Automated. \"Crowdsourcing\" is evolving as a distributed problem-solving and business production model in recent years [89]. An example of crowdsourcing website is Amazon Mechanical Turk. Today, many MRC datasets are posed by the distributed workforce on such crowdsourcing websites. The \"Expert\" generation method means that question and answer pairs in the dataset are generated by people with professional knowledge in some fields. For example, in the ARC dataset [72], there are 7787 science questions covered by US elementary and middle schools. The \"Automated\" generation method means that question and answer pairs are automatically generated based on corpus, such as many cloze datasets.  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Year** & **Datasets** & **Corpus Size** & **\#Train Corpus** & **\#Dev Corpus** & **\#Test Corpus** & **Unit of Corpus** \\ \hline
2016 & WikiReading [56] & 4.7M & N/A & N/A & N/A & Article \\
2016 & SQuAD 1.1 [19] & 536 & 442 & 48 & 46 & Article \\
2018 & SQuAD 2.0 [66] & 505 & 442 & 35 & 28 & Article \\
2016 & BookTest [57] & 14,062 & N/A & N/A & N/A & Book \\
2017 & COMICS [33] & 3948 & N/A & N/A & N/A & Book \\
2016 & Facebook CBT [28] & 108 & 98 & 5 & 5 & Book \\
2019 & DREAM [81] & 6444 & 3869 & 1288 & 1287 & Dialogue \\
2016 & NewsQA [54] & 1,010,916 & 909,824 & 50,546 & 50,546 & Document \\
2017 & TriviaQA-Web [59] & 662,659 & 528,979 & 68,621 & 65,059 & Document \\
2015 & Daily Mail [19] & 219,506 & 196,961 & 12,148 & 10,397 & Document \\
2017 & TriviaQA-Wiki [59] & 138,538 & 110,648 & 14,229 & 13,661 & Document \\
2018 & ReviewQA [25] & 100,000 & 90,000 & N/A & 10,000 & Document \\
2015 & CNN [19] & 92,579 & 90,266 & 1220 & 1093 & Document \\
2017 & NarrativeQA [63] & 1572 & 1102 & 115 & 355 & Document \\
2017 & TQA [31] & 1076 & 666 & 200 & 210 & Lesson \\
2016 & MovieQA [32] & 548 & 362 & 77 & 109 & Movie \\
2016 & Google MC-AFP [58] & 1,742,618 & 1,727,423 & 7602 & 7593 & Passage \\
2016 & Who-did-What [48] & 147,786 & 127,786 & 10,000 & 10,000 & Passage \\
2017 & SearchQA [62] & 140,461 & 99,820 & 13,393 & 27,248 & Passage \\
2018 & ReCoRD [70] & 80,121 & 65,709 & 7133 & 7279 & Passage \\
2017 & Quasar-T [61] & 43,012 & 37,012 & 3000 & 3000 & Passage \\
2017 & Quasar-S [61] & 37,362 & 31,049 & 3174 & 3139 & Passage \\
2017 & RACE [60] & 27,933 & 25,137 & 1389 & 1407 & Passage \\
2018 & SciTail [74] & 27,026 & 23,596 & 1304 & 2126 & Passage \\
2016 & LAMBADA [55] & 12,684 & 2662 & 4869 & 5153 & Passage \\
2018 & CoQA [47] & 8399 & 7199 & 500 & 700 & Passage \\
2018 & CLOTH [69] & 7131 & 5513 & 805 & 813 & Passage \\
2017 & Qangaroo-W [49] & 51,318 & 43,738 & 5129 & 2451 & Passage \\
2019 & DROP [50] & 6735 & 5565 & 582 & 588 & Passage \\
2017 & Qangaroo-M [49] & 2508 & 1620 & 342 & 546 & Passage \\
2018 & RecipeQA [34] & 19,779 & 15,847 & 1963 & 1969 & Recipe \\
2015 & WikQA [52] & 29,258 & 20,360 & 2733 & 6165 & Sentence \\
2013 & MCTest-mc500 [15] & 500 & 300 & 50 & 150 & Story \\
2013 & MCTest-mc160 [15] & 160 & 70 & 30 & 60 & Story \\
2018 & QuAC [65] & 8845 & 6843 & 1000 & 1002 & Unique section \\
2019 & ShARC [38] & 32,436 & 21,890 & 2270 & 8276 & Utterance \\
2019 & Natural Questions [80] & 323,045 & 307,373 & 7830 & 7842 & Wikipedia Page \\ \hline \hline \end{tabular}
\end{table}
Table 5: The corpus size of machine reading comprehension datasets.