# A Survey on Machine Reading Comprehension--Tasks, Evaluation Metrics and Benchmark Datasets
## 4 Benchmark Dataset
### Characteristics of Datasets
#### 4.9.2 MRC with Unanswerable Questions  
The existing MRC datasets often lack training sets for unanswerable questions, which weaken the robustness of the MRC systems. As a result, when the MRC models answer unanswerable questions, the models always try to give a most likely answer, rather than refuse to answer these unanswered questions. In this way, no matter how the model answers, the answers must be wrong.  
To solve this problem, the researchers proposed many MRC datasets with unanswerable questions which were more challenging. Among the datasets collected by us, the datasets that contain unanswerable questions include: SQuAD 2.0, MS MARCO [39], Natural Questions [80] and NewsQA [54]. We will give a detailed description of these datasets in section in Section 4.10.  
\begin{table}
\begin{tabular}{l l l} \hline \hline
**Year** & **Datasets** & **Characteristics** \\ \hline
2015 & WikiQA [52] & With Unanswerable Questions \\
2018 & SQuAD 2.0 [66] & With Unanswerable Questions \\
2019 & Natural Question [80] & With Unanswerable Questions \\
2016 & MS MARCO [39] & With Unanswerable Questions; Multi-hop MRC \\
2018 & DuoRC [68] & With Paraphrased Paragraph; Require Commonsense (World knowledge); \\
2016 & Who-did-What [48] & With Paraphrased Paragraph; Complex Reasoning \\
2018 & ARC [72] & Require Commonsense (World knowledge); Complex Reasoning \\
2018 & MCScript [78] & Require Commonsense (World knowledge) \\
2018 & OpenBookQA [73] & Require Commonsense (World knowledge) \\
2018 & RecORD [70] & Require Commonsense (World knowledge) \\
2019 & CommonseQA [82] & Require Commonsense (World knowledge) \\
2016 & WikiReading [56] & Require Commonsense (External knowledge); Large Scale Dataset \\
2016 & WikiMovies [53] & Require Commonsense (External knowledge); Domain-specific \\
2016 & MovieQA [32] & Multi-Modal MRC \\
2017 & COMICS [53] & Multi-Modal MRC \\
2017 & TQA [31] & Multi-Modal MRC \\
2018 & RecipeQA [34] & Multi-Modal MRC \\
2018 & HotpotQA [67] & Multi-hop MRC; Complex Reasoning \\
2017 & NarrativeQA [63] & Multi-hop MRC; Complex Reasoning \\
2017 & Qangaroo [49] & Multi-hop MRC \\
2018 & MultiRC [75] & Multi-hop MRC MRC \\
2015 & CNN/Daily Mail [19] & Large-scale Dataset \\
2016 & BookTest [57] & Large-scale Dataset \\
2013 & MCTest [15] & For Open-domain QA \\
2015 & CuratedTREC [51] & For Open-domain QA \\
2017 & Quasar [61] & For Open-domain QA \\
2017 & SearchQA [62] & For Open-domain QA \\
2017 & SciQ [64] & Domain-specific \\
2018 & CliCR [71] & Domain-specific \\
2018 & PaperQA([PERSON] et al.) [77] & Domain-specific \\
2018 & PaperQA([PERSON] et al.) [76] & Domain-specific \\
2018 & ReviewQA [25] & Domain-specific \\
2018 & SciTail [74] & Domain-specific \\
2019 & DROP [50] & Complex Reasoning \\
2016 & Facebook CBT [28] & Complex Reasoning \\
2016 & Google MC-AFP & Complex Reasoning \\
2016 & LAMBADA [55] & Complex Reasoning \\
2016 & NewsQA [54] & Complex Reasoning \\
2016 & SQuAD 1.1 [19] & Complex Reasoning \\
2017 & RACE [60] & Complex Reasoning \\
2017 & TriviaQA [59] & Complex Reasoning \\
2018 & CLOTH [69] & Complex Reasoning \\
2018 & ProPara [79] & Complex Reasoning \\
2019 & DREAM [81] & Conversational MRC; Require Commonsense (World knowledge) \\
2018 & CoQA [47] & Conversational MRC; With Unanswerable Questions \\
2018 & QuAC [65] & Conversational MRC; With Unanswerable Questions \\
2019 & ShARC [38] & Conversational MRC \\ \hline \hline \end{tabular}
\end{table}
Table 10: The characteristics of each MRC dataset.