# Neural Machine Reading Comprehension: Methods and Trends
## 4 Datasets and Evaluation Metrics
### Evaluation Metrics  
For different MRC tasks, there are various evaluation metrics. To evaluate cloze tests and multiple-choice tasks, the most common metric is accuracy. In terms of span extraction, exact match (EM), a variant of accuracy, and F1 score are computed to measure model performance. Considering that answers for free answering tasks are not limited to the original context, ROUGE-L and BLEU are widely used. In the following part, we will give detailed descriptions of these evaluation metrics. * Accuracy with respect to ground-truth answers is usually applied to evaluate cloze tests and multiple-choice tasks. When given a question set \(Q=\{Q_{1},Q_{2},\cdots Q_{m}\}\) with \(m\) questions, if the model correctly predicts answers for \(n\) questions, then accuracy is calculated as follows: \[\text{Accuracy}=\frac{n}{m}.\] (10)  
Exact match is a variant of accuracy that evaluates whether a predicted answer span matches the ground-truth sequence exactly or not. If the predicted answer is equal to the gold answer, the EM value will be 1, otherwise 0. It can also be calculated by the above equation. * F1 Score  
F1 score is a common metric in classification tasks. In terms of MRC, both candidate and reference answers are treated as bags of tokens and true positive (TP), false positive (FP), true negative (TN), and false negative (FN) are denoted as shown in Table 8.  
Then precision and recall are computed as below:  
\[\text{precision}=\frac{\text{TP}}{\text{TP}+\text{FP}}\, \tag{11}\]  
\[\text{recall}=\frac{\text{TP}}{\text{TP}+\text{FN}}. \tag{12}\]  
F1 score, also known as balanced F score, is the harmonic average of precision and recall:  
\[\text{F1}=\frac{2\times P\times R}{P+R}\, \tag{13}\]  
where \(P\) denotes precision while \(R\) is recall. Compared to EM, this metric loosely measures the average overlap between the prediction and the ground-truth answer. \begin{table}
\begin{tabular}{c c c} \hline \hline \multicolumn{3}{c}{**Tokens in Reference**} & **Tokens Not in Reference** \\ \hline tokens in candidate & TP & FP \\ tokens not in candidate & FN & TN \\ \hline \hline \end{tabular}
\end{table} 
Table 8: The definition of true positive (TP), true negative (TN), false positive (FP), false negative (FN). * ROUGE-L  
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is an evaluation metric initially developed for automatic summarization, proposed by [PERSON] and [PERSON] [78]. It evaluates the quality of a summary by counting the amount of overlap between model-generated and ground-truth summaries. There are various ROUGE measures, such as ROUGE-N, ROUGE-L, ROUGE-W, and ROUGE-S, for different evaluation requirements, among which ROUGE-L is widely used in MRC tasks with the use of free answering. Unlike other metrics, such as EM or accuracy, ROUGE-L is more flexible, mainly measuring similarities between gold answers and predicted answers. The \"L\" in ROUGE-L denotes longest common subsequence (LCS) and ROUGE-L can be computed as follows:  
\[R_{\text{Lcs}}=\frac{LCS(X,Y)}{m}, \tag{14}\]  
\[P_{\text{Lcs}}=\frac{LCS(X,Y)}{n}, \tag{15}\]  
\[F_{\text{Lcs}}=\frac{(1+\beta)^{2}R_{\text{Lcs}}P_{\text{Lcs}}}{R_{\text{Lcs}}+ \beta^{2}P_{\text{Lcs}}}, \tag{16}\]  
where \(X\) is ground-truth answer with \(m\) tokens, \(Y\) is model-generated answer with \(n\) tokens, \(\text{LCS}(X,Y)\) denotes the length of the longest common subsequence of \(X\) and \(Y\), and \(\beta\) is a parameter to control the importance of precision \(P_{\text{Lcs}}\) and recall \(R_{\text{Lcs}}\). Using ROUGE-L to evaluate the performance of an MRC model does not require predicted answers to be consecutive subsequences of ground truth, whereas more token overlap contributes to higher ROUGE-L scores. However, the length of candidate answers influences the value of ROUGE-L.  
* BLEU  
BLEU (Bilingual Evaluation Understudy), proposed by [PERSON] et al. [79], is widely used to evaluate translation performance. When adapted to MRC tasks, BLEU score measures the similarity between predicted answers and ground truth.