# A Review on Medical Textual Question Answering Systems Based on Deep Learning Approaches
## 4 Evaluation Metrics and Datasets
### Datasets  
In this subsection, we discuss the common datasets used for QA task in the medical domain. Our review excludes datasets used in the open domain QA, where any kind of question can find an answer. We only focus on the publicly available medical datasets used by health workers to find relevant answers that assist them in their profession.  
1. MedQuAD: Medical Question Answering Dataset  
This dataset was created from 12 National Institutes of Health (NIH) websites, such as cancer.gov, niddk.nih.gov, MedlinePlus Health Topics, GARD, etc. It contains 47,457 medical question-answer pairs, and covers 37 question types, such as Treatment, Diagnosis, and Side Effects, associated with diseases, drugs, and other medical entities [63]. The experiments of [63] were conducted on this dataset, and the experimental results show that the inter-annotator agreement through \(F1\) score achieved 88.5% agreement on the four categories and a 94.3% agreement when the categories are reduced to two.
2. EmrQA: Electronic Medical Records for Question Answering  
EmrQA is a dataset that focuses on the medical domain [76]. It is worth mentioning that the EmrQA dataset is leveraged by utilizing the existing expert annotations on clinical notes for different NLP tasks in the community-shared i2b2 dataset. The EmrQA dataset  
Figure 15: The evaluation for different classification methods in [75].  
has one million question-logical forms and 400,000+ question-answer evidence pairs. The model proposed in [76] was evaluated on the dataset EmrQA. The experimental results show that the proposed model can achieve 60.6% for \(F1\) and 59.2% for \(EM\) (Exact Match).
3. QA4 MRE: Question Answering for Machine Reading Evaluation This dataset contains three topics: Climate change, Aids, and Music & Society [13; 77]. Each topic includes four reading tests. Each reading test consists of one single document, with 10 questions and a set of five choices per question. In total, there are 16 test documents (four documents for every three topics), 160 questions (10 questions for each document), with 800 choices (5 for each question). Test documents and questions were made available in English, German, Italian, Romanian, and Spanish. These materials were the same in all languages, created using parallel translations. The evaluation performed on the QA4 MRE dataset is presented in [77], and the experimental result showed that 53% of the questions were answered with the correct candidate.
4. cMedQA: Chinese Medical Question and Answers This is the dataset for Chinese community MQA [60]. The cMedQA dataset facilitates choosing QA pairs from some real-world online health ([[http://www.xywy.com/](http://www.xywy.com/)]([http://www.xywy.com/](http://www.xywy.com/)), accessed on 15 December 2020) and wellness communities, such as DingXiangYuan and XunYi-WenYao. The dataset consists of 101,743 QA pairs, where the sentences were split into individual characters. The vocabulary has a total of 4979 tokens. In [60], different models such as SingleCNN, biLSTM, and Multi-CNN were compared on the cMedQA dataset. The results show that these models can achieve 64.05%, 63.20%, and 64.75% accuracy, respectively.
5. MASH-QA: Multiple Answer Spans Healthcare Question Answering MASH-QA is a large-scale dataset for QA, with many answers coming from multiple spans within a long document. The dataset consists of over 35,000 QA pairs and is based on questions and knowledge articles from the consumer health domain, where the questions are generally non-factoid in nature and cannot be answered using just a few words [78]. The experimental results in [78] show that using models of DrQA Reader, BiDAF, BERT, SpanBERT, XLnet and MultiCo, on the MASH-QA dataset, the \(F1\) are 18.92%, 23.19%, 27.93%, 30.61%, 56.46%, and 64.94% and \(EM\) are 1.82%, 2.42%, 3.95%, 5.62%, 22.78%, and 29.49% respectively. Table 4 gives the details of different QA datasets used in the healthcare domain.