# A Review on Medical Textual Question Answering Systems Based on Deep Learning Approaches
## 4 Evaluation Metrics and Datasets
### Evaluation Metrics  
Evaluation is one of the essential dimensions in QA, as it assesses and compares the answers to measure the performances of QASs [74]. Much effort has been put into addressing the problem of the performance evaluation of IR systems [40]. Based on these efforts, there are two approaches to the performance evaluation of QASs: system-centered and user-centered. Evaluation methods play an important role in the QA system. With the rapid development of QA methods, reliable evaluation metrics are needed to compare these imple  
Figure 14: Hybrid architecture of CNN and BiLSTM. mentations [34]. The metrics often used in QA are \(F1\) and accuracy [38]. For any given datapoint to be assessed, there must be two categories: a segment that is correctly selected (true positive) or not correctly selected (false negative), and a segment that is not correctly selected (false positive) or incorrectly not selected (true negative). Equation (23) shows how _Accuracy_ is calculated  
\[Accuracy=\frac{TP+TN}{TP+FP+FN+TN} \tag{23}\]  
where \(TP\), \(FP\), \(TN\), and \(FN\) are representing true positive, false positive, true negative, and false negative, respectively. It can be observed for the QA systems metric in evaluation, that the system gives the correct answer when a fact question is asked, and vice versa. Therefore, the system can be found to have a high true negative rate. For example, the system may have high computational accuracy, but it is not meaningful. To solve this problem, the \(F\)-measure is adopted, which is based on two metrics: Precision and Recall. The first is the percentage of selected answers among correct answers, while the second is the reverse measure. That means that Recall is the percentage of correct answers selected. When using Precision and Recall, there is no longer a real fact that the rate of negative answers is high. Table 3 summarizes the concepts of the three evaluation metrics: _Accuracy_, _Precision_, and _Recall_. \[Precision=\frac{TP}{TP+FP}(A1) \tag{24}\]  
\[Recall=\frac{TP}{TP+FN}(A2) \tag{25}\]  
Equations (24) and (25) give a fundamental understanding of the researchers' trade-off to conduct in each measure in looking for the best metrics to evaluate their systems. Most of the fact QA systems should use Equation (25) as a measured metric, since it does not matter how high the false positive rates are; the result will be good if there are high true positive rates. However, for list or definition QA systems, Equation (24) is assumed to function better. To balance this trade-off, the \(F\)-measure is presented. \[F_{1}=\frac{2(Precision\cdot Recall)}{Precision+Recall} \tag{26}\]  
Equation (26) implements a weighted approach to assessment of the Precision and Recall trade-off. Some metrics can be used to evaluate QA systems, such as Mean Average Precision (\(MAP\)) presented in Equation (27), and Mean Reciprocal Rank (\(MRR\)) shown in Equation (29) used to calculate the answer relevance. The two are mainly used in IR paradigms. \[MAP=\frac{\sum_{Q}^{q=1}AveP(q)}{Q} \tag{27}\]  
where \(Q\) stands for queries and the average precision (\(AveP\)) can be expressed as  
\[AveP=\frac{\sum_{N}^{k=1}P(k)\times rel(q)}{|\{relevant\}|} \tag{28}\] \begin{table}
\begin{tabular}{c c c} \hline \hline
**Actual** & **Predicted** & **Positive** & **Negative** \\ \hline Positive & True Positive & False Positive \\ Negative & False Negative & True Negative \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Accuracy, Precision and Recall. \[MRR=\frac{1}{n}\sum_{i=1}^{N}RR_{(q_{i})} \tag{29}\]  
Evaluation metrics can also be categorized based on the types of question. For factoid questions, they consider the highest probability answer as the exact answer. Three evaluation metrics used for factoid questions are: Strict Accuracy, Lenient Accuracy, and MRR. For the list questions, they set a threshold, then all the predictions above that threshold are considered as the list of answers to the question.