# A Survey on Machine Reading Comprehension--Tasks, Evaluation Metrics and Benchmark Datasets
## 3 Evaluation Metrics
### Statistics of Evaluation Metrics
\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Year** & **MRC Tasks** & **Metric 1** & **Metric 2** & **Metric 3** & **Metric 4** \\ \hline
2013 & MCTC [15] & Accuracy & N/A & N/A & N/A \\
2015 & CNN/Daily Mail [19] & Accuracy & N/A & N/A & N/A \\
2015 & Curated/FRCE [51] & Exact Match & N/A & N/A & N/A \\
2015 & WikQA [2] & Question-level Precision & Question-level Recall & Question-level F1 & N/A \\
2016 & RookTest [57] & Accuracy & N/A & N/A & N/A \\
2016 & Facebook-CIF [28] & Accuracy on Named Entities & Accuracy on Common Nouns & Accuracy on Perpositions \\
2016 & Google MC-CIF [58] & Accuracy & N/A & N/A & N/A \\
2016 & LAMBADA [53] & Accuracy & N/A & N/A & N/A \\
2016 & MovieQA [2] & Accuracy of Video Clips & Accuracy of Plots and Subtitles & N/A & N/A \\
2016 & MSARC [39] & Rougel- & BLEU-1 & N/A & N/A \\
2016 & NewsQA [54] & Exact Match & Token-level F1 & N/A & N/A \\
2016 & SQuAD [11] & Token-level F1 & Exact Match & N/A & N/A \\
2016 & Who-did-What [48] & Accuracy & N/A & N/A & N/A \\
2016 & WikidNeigh [53] & Accuracy & N/A & N/A & N/A \\
2016 & WikidNeigh [56] & Question level F1 & N/A & N/A & N/A \\
2017 & COMMICS-CI [33] & Accuracy of Text Cloze & Accuracy of Visual Cloze & N/A & N/A \\
2017 & COMMICS-CI [33] & Accuracy of Coherence & N/A & N/A & N/A \\
2017 & NarrativeQA [63] & ROUGE-L & BLEU-1 & BLEU-4 & Meteor \\
2017 & QangarooM [49] & Accuracy & N/A & N/A & N/A \\
2017 & Qangaroo-W [49] & Accuracy & N/A & N/A & N/A \\
2017 & Quasar-S [61] & Accuracy & N/A & N/A & N/A \\
2017 & Quasar-T [61] & Exact Match & Token-level F1 & N/A & N/A \\
2017 & RACE [60] & Accuracy & N/A & N/A & N/A \\
2017 & SciQ [4] & Accuracy & N/A & N/A & N/A \\
2017 & SearchQ [62] & F1 score (for \(n\)-gram) & Accuracy & N/A & N/A \\
2017 & TQA [31] & Accuracy of All & Accuracy of Diagram & N/A & N/A \\
2017 & TriviaQA-Wab [59] & Exact Match & Question-level F1 & Verified-EM & Verified-FI \\
2017 & TriviaQA-Wab [59] & Exact Match & Document-level F1 & Verified-EM & Verified-FI \\
2018 & ARC C [72] & Accuracy & N/A & N/A & N/A \\
2018 & Click [71] & Exact Match & Token-level F1 & BLEU-4 & BLEU-4 \\
2018 & CLQIM [69] & Accuracy & N/A & N/A & N/A \\
2018 & CQQA [47] & Token-level F1 & F1 out of domain & F1 in domain & N/A \\
2018 & DuokC-P [58] & Accuracy & Token-level F1 & N/A & N/A \\
2018 & DuokC-S [58] & Accuracy & Token-level F1 & N/A & N/A \\
2018 & HoptopQA-D [67] & EM of Answer & F1 of Answer (Token-level) & EM of Supportions & F1 of Supportions \\
2018 & HotpotQA-D [67] & EM of Answer & F1 of Answer (Token-level) & EM of Supportions & F1 of Supportions \\
2018 & MCScript [78] & Accuracy & N/A & N/A & N/A \\
2018 & Music [53] & F1m & Exact Match & N/A & N/A \\
2018 & OpenBookQA [73] & Accuracy & N/A & N/A & N/A \\
2018 & PaperQA([PERSON] et al.) [7] & F1 & N/A & N/A & N/A \\
2018 & PaperQA-L5 [6] & Accuracy & N/A & N/A & N/A \\
2018 & PaperQA-T [79] & Accuracy & N/A & N/A & N/A \\
2018 & Profa [59] & Accuracy & N/A & N/A & N/A \\
2018 & QuAC [65] & Token-level F1 & HEQ-Q & HEQ-D & N/A \\
2018 & ReebQA-C [34] & Accuracy of Textual Cloze & Accuracy of Visual Cloze & N/A & N/A \\
2018 & ReebQA-C [34] & Accuracy-VO & Accuracy-VC & N/A & N/A \\
2018 & ReCoRD [70] & Exact Match & Token-level F1 & N/A & N/A \\
2018 & ReviewQA [2] & Accuracy & N/A & N/A & N/A \\
2018 & SciNLP [4] & Accuracy & N/A & N/A & N/A \\
2018 & SGUAD 20 [66] & Token-level F1 & EM & N/A & N/A \\
2019 & CommonSenseQA [54] & Accuracy & N/A & N/A & N/A \\
2019 & DREAM [81] & Accuracy & N/A & N/A & N/A \\
2019 & DROP [50] & EM & Token-level F1 & N/A & N/A \\
2019 & Natural Questions-Long [80] & Precision & Recall & N/A & N/A \\
2019 & Natural Questions-Short [80] & Precision & Recall & F1 & N/A \\
2019 & ShARC [58] & Micro Accuracy & Macro Accuracy & BLEU-1 & BLEU-4 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Evaluation metrics of different machine reading comprehension tasks.