SlowGuess/ACL_2020_OCR huggingface dataset:
Xiang Yue
Bernal Jimenez Gutierrez
Huan Sun
{yue.149, jimenezgutierrez.1, sun.397}@osu.edu
Machine reading comprehension has made great progress in recent years owing to large-scale annotated datasets. In the clinical domain, however, creating such datasets is quite difficult due to the domain expertise required for annotation. Recently, Pampari et al. (2018) tackled this issue by using expert-annotated question templates and existing i2b2 annotations to create emrQA, the first large-scale dataset for question answering (QA) based on clinical notes. In this paper, we provide an in-depth analysis of this dataset and the clinical reading comprehension (CliniRC) task. From our qualitative analysis, we find that (i) emrQA answers are often incomplete, and (ii) emrQA questions are often answerable without using domain knowledge. From our quantitative experiments, surprising results include that (iii) using a small sampled subset  $(5\% - 20\%)$ , we can obtain roughly equal performance compared to the model trained on the entire dataset, (iv) this performance is close to human expert's performance, and (v) BERT models do not beat the best performing base model. Following our analysis of the emrQA, we further explore two desired aspects of CliniRC systems: the ability to utilize clinical domain knowledge and to generalize to unseen questions and contexts. We argue that both should be considered when creating future datasets.
Medical professionals often query over clinical notes in Electronic Medical Records (EMRs) to find information that can support their decision making (Demner-Fushman et al., 2009; Rosenbloom et al., 2011; Wang et al., 2018). One way to facilitate such information seeking activities is to build a natural language question answering (QA) system that can extract precise answers from clinical notes (Cairns et al., 2011; Cao et al., 2011; Wren, 2011; Abacha and Demner-Fushman, 2016, 2019).
RECORD #992321, Date: 2145-09-22
Context: ... For HTN control, pt was given HCTZ and lopressor which sufficiently controlled his BP. Pt was sent home on HCTZ 25mg daily and atenolol 50mg daily.
ADDITIONAL COMMENTS: 1.) Take hydrochlorothiazide  $25\mathrm{mg}$  daily and atenolol  $50\mathrm{mg}$  daily for your blood pressure. You should also take aspirin  $81\mathrm{mg}$  daily.
Question: Why has the patient been prescribed hctz?
Answer: For HTN control, pt was given HCTZ and lopressor which sufficiently
Question: What was the dosage prescribed of hydrochlorothiazide?
Answer: ADDITIONAL COMMENTS: 1.) Take hydrochlorothiazide  $25\mathrm{mg}$  daily and atenolol  $50\mathrm{mg}$  daily for your
Figure 1: Examples from the emrQA dataset: Part of a clinical note as context and 2 question-answer pairs. Due to the original emrQA generation issues, oftentimes answers are incomplete or contain irrelevant parts to the questions (the underlined parts are what we think the most relevant to the questions).
Machine reading comprehension (RC) aims to automatically answer questions based on a given document or text corpus and has drawn wide attention in recent years. Many neural models (Cheng et al., 2016; Wang et al., 2017; Wang and Jiang, 2017; Seo et al., 2017; Chen et al., 2017; Devlin et al., 2019) have achieved very promising results on this task, owing to large-scale QA datasets (Hermann et al., 2015; Rajpurkar et al., 2016; Trischler et al., 2017; Joshi et al., 2017; Yang et al., 2018). Unfortunately, clinical reading comprehension (CliniRC) has not observed as much progress due to the lack of such QA datasets.
In order to create QA pairs on clinical texts, annotators must have considerable medical expertise and data handling must be specifically designed to address ethical issues and privacy concerns. Due to these requirements, using crowdsourcing like in the open domain to create large-scale clinical QA
1Our code is available at https://github.com/xiangyue9607/CliniRC.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4474-4486 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
datasets becomes highly impractical (Wei et al., 2018).
Recently, Pampari et al. (2018) found a smart way to tackle this issue and created emrQA, the first large-scale QA dataset on clinical texts. Instead of relying on crowdsourcing, emrQA was semiautomatically generated based on annotated question templates and existing annotations from the n2c2 (previously called i2b2) challenge datasets<sup>2</sup>. Example QA pairs from the dataset are shown in Figure 1.
In this paper, we aim to gain a deep understanding of the CliniRC task and conduct a thorough analysis of the emrQA dataset. We first explore the dataset directly by carrying out a meticulous qualitative analysis on randomly-sampled QA pairs and we find that: 1) Many answers in the emrQA dataset are incomplete and hence are hard to read and ineffective for training (§3.1). 2) Many questions are simple: More than  $96\%$  of the examples contain the same key phrases in both questions and answers. Though Pampari et al. (2018) claims that  $39\%$  of the questions may need knowledge to answer, our error analysis suggests only a very small portion of the errors  $(2\%)$  made by a state-of-the-art reader might be due to missing external domain knowledge (§3.2).
Following our qualitative analysis of the emrQA dataset, we conduct a comprehensive quantitative analysis based on state-of-the-art readers and BERT models (BERT-base (Devlin et al., 2019) as well as its biomedical and clinical versions: BioBERT (Lee et al., 2019) and ClinicalBERT (Alsentzer et al., 2019)) to understand how different systems behave on the emrQA dataset. Surprising results include: 1) Using a small sampled subset  $(5\% -20\%)$ , we can obtain roughly equal performance compared to the model trained on the entire dataset, suggesting that many examples in the dataset are redundant ( $\S 4.1$ ). 2) The performance of the best base model is close to the human expert's performance<sup>3</sup> ( $\S 4.2$ ). 3) The performance of BERT models is around  $1\% -5\%$  worse than the best performing base model ( $\S 4.3$ ).
After completing our analysis of the dataset, we explore two potential needs for systems doing CiniRC: 1) The need to represent and use clinical domain knowledge effectively (§5.1) and 2) the need to generalize to unseen questions and contexts (§5.2). To investigate the first one, we analyze sev
Medication	Relation
# Question	222,957	904,592
# Context	261	423
# Question Template	80	139
Question: avg. tokens	8.00	7.91
Answers: avg. tokens	9.47	10.41
Context: avg. tokens	1062.66	889.23
Table 1: Statistics of two major subsets, Medication and Relation, of the emrQA dataset.
eral types of clinical questions that require domain knowledge and can frequently appear in the real clinical setting. We also carry out an experiment showing that adding knowledge explicitly yields around  $5\%$  increase in F1 over the base model when tested on samples that we created by altering the original questions to involve semantic relations. To study generalizability, we ask medical experts to create new questions based on the unseen clinical notes from MIMIC-III (Johnson et al., 2016), a freely accessible critical care database. We find that the performance of the best model trained on emrQA drops by  $40\%$  under this new setting, showing how critical it is for us to develop more robust and generalizable models for the CliniRC task.
In summary, given our analysis of the emrQA dataset and the task in general, we conclude that future work still needs to create better datasets to advance CliniRC. Such datasets should be not only large-scale, but also less noisy, more diverse, and allow researchers to directly evaluate a system's ability to encode domain knowledge and to generalize to new questions and contexts.
2 Overview of the emrQA dataset
Similar to the open-domain reading comprehension task, the Clinical Reading Comprehension (CliniRC) task is defined as follows:
Definition 2.1. Given a patient's clinical note (context)  $C = \{c_1, \dots, c_n\}$  and a question  $Q = \{t_1, \dots, t_m\}$ , the CiniRC task aims to extract a continuous span  $A = \{c_i, c_{i+1}, \dots, c_{i+k}\} (1 \leq i \leq i + k \leq n)$  from the context as the answer, where  $c_i, t_j$  are tokens.
The emrQA dataset (Pampari et al., 2018) was semi-automatically generated from expert-annotated question templates and existing i2b2 annotations. More specifically, clinical question templates were first created by human experts. Then, manual annotations from the medication information extraction, relation learning, and coreference
$^{2}$ https://portal.dbmi.hms.harvard.edu/projects/n2c2-nlp/  $^{3}$ Which is obtained by comparing emrQA answers to answers created by our medical experts on sampled QA pairs.
Question Template
Has the patient ever been on I medication I?
Existing i2b2 Annotation
<Medication="Flagyl", Line Index = 128>
Generated Question
Has the patient ever been on Flagyl?
Generated Answer
Flagyl. By discharge, the patient was afebrile (line 128)
Figure 2: An example to illustrate how emrQA generates QA pairs.
resolution i2b2 challenges were re-framed into answers for the question templates. After linking question templates to i2b2 annotations, the gold annotation entities were used to both replace placeholders in the question templates and extract the sentence around them as answers. An example of this generation process can be seen in Figure 2.
The emrQA dataset contains 5 subsets: Medication, Relation, Heart Disease, Obesity and Smoking, which were generated from 5 i2b2 challenge datasets respectively. The answer format in each dataset is different. For the Obesity and Smoking datasets, answers are categorized into 7 classes and the task is to predict the question's class based on the context. For the Medication, Relation, and Heart Disease datasets, answers are usually short snippets from the text accompanied by a longer span around it which we refer to as an evidence. The short snippet is a single entity or multiple entities while the evidence contains the entire line around those entities in the clinical note. For questions that cannot be answered via entities, only the evidence is provided as an answer. Given that some questions do not have short answers and that entire evidence spans are usually important for supporting clinical decision making (Demner-Fushman et al., 2009), we treat the answer evidence<sup>4</sup> as our answer just as is done in (Pampari et al., 2018).
In this work, we mainly focus on the Medication and Relation datasets because (1) they make up  $80\%$  of the entire emrQA dataset and (2) their format is consistent with the span extraction task, which is more challenging and meaningful for clinical decision making support. We filter the answers whose lengths (number of tokens) are more than 20. The detailed statistics of the two datasets are shown in Table 1.
Metric	Medication	Relation
Quality Score	3.92	4.75
EM	26.0	92.0
F1	74.7	95.4
Table 2: An estimate of the quality of answers in the Medication and Relation datasets based on the analysis of our randomly sampled 50 questions for each dataset. Quality scores are the average of two human annotators' (maximum: 5). EM and F1 scores are calculated between human-labeled answers v.s. emrQA answers.
3 In-depth Qualitative Analysis
In this section, we carry out an in-depth analysis of the emrQA dataset. We aim to examine (1) the quality and (2) level of difficulty for the generated QA pairs in the emrQA dataset.
3.1 How clean are the emrQA answers?
Since the emrQA dataset was created via a generation framework unlike human-labeled or crowdsourcing datasets, the quality of the datasets remains largely unknown. In order to use this dataset to explore the CliniRC task, it is essential to determine whether it is meaningful.
In order to do this, we randomly sample 50 QA pairs from the Medication and the Relation datasets respectively. Since some questions share the same answer due to automatic generation, we make sure all the samples have different answers.
Since the questions were generated from expert created templates, most of them are human-readable and unambiguous. We therefore mainly focus on evaluating answer quality. We ask two human experts to score each answer from 1 to 5 depending on the relevance of the answer to the question (1: irrelevant or incorrect; 2: missing key parts; 3: contains key parts but is not human-readable or contains many irrelevant parts; 4: contains key parts and is only missing a few parts or has a few irrelevant extra segments; 5: perfect answer). We also ask human annotators to label the gold answers and then calculate the Exact Match (EM) and F1 score (F1) of the emrQA answers v.s. human gold answers. The answer quality score, EM and F1 in both datasets, are shown in Table 2.
The scores of the Medication dataset are low since most of the answers are broken sentences or contain unnecessary segments. For instance, in the Figure 2 example, the correct answer should be "Clindamycin was changed to Flagyl", how
4 For simplicity, we use "answer" directly henceforth.
Error Type	Question	emrQA Answers	Prediction	Error Ratio
Medication	Relation
Span mismatch - include key info	Does she have a history of known drug allergies?	ALLERGIES:
He had no known drug allergies	He had no known drug allergies	78%	66%
Span mismatch - miss key info	What is the current dose of lasix?	MEDS: K-Dur 20 BID, Nexium 20, lasix 160 BID	BID	4%	0%
Ambiguous questions	What is the patient's low history?	At the time of discharge, her potassium had been low despite repletion	11) Low grade,anemia	8%	4%
Incorrect
golds	What is the patient's incisions status?	Wash incisions with warm water and gentle soap	Do not apply lotions, creams, ointments or powders to incision	2%	2%
False negatives	Is there a mention of fluid in the record?	There is some fluid, or mucosal thickening in the ethmoid and sphenoid sinuses	The amount of fluid layering at the apices and the pleural spaces appear slightly decreased	2%	18%
May need external knowledge	What treatment has the patient had for his CAD?	CAD s/p CABG 2003 s/p	Pt's vancomycin was stopped after 14 days of treatment	2%	2%
Others	Is the patient's right hand ganglion cyst well-controlled?	right hand ganglion cyst removed	x 3 right hand ganglion cyst	4%	8%
Table 3: Error analysis on 50 sampled questions from the Medication and Relation dev sets respectively. Example question, ground truth and prediction from either Medication or Relation are given for each type of error.
ever, the emrQA answer misses important parts "Clindamycin was changed to" and contains irrelevant parts "By discharge, the patient was afebrile". These issues are common in the Medication dataset and make it difficult to train a good system. To understand why the generated answers contain such noise, we explored the "i2b2 2009 Medication" challenge dataset which was used to create these QA pairs. We found that most documents in this dataset contain many complete sentences split into separate lines. Since the i2b2 annotation are token based and the emrQA obtains full lines around the token as evidence spans, these lines often end up being broken sentences. We tried to relabel the answers with existing sentence segmentation tools and heuristic measures but found that it is very challenging to obtain concise and complete text spans as answers.
Compared with the Medication dataset, the answer quality of the Relation dataset is much better. In most cases, the answers are complete and meaningful sentences with no unnecessary parts.
3.2 How challenging are the emrQA pairs?
Another observation from the 50 samples is that  $96\%$  of the answers in the Medication dataset and  $100\%$  of the answers in the Relation dataset contain the key phrase in the question. This is due to the generation procedure illustrated in Figure 2. In this
example, the key phrase or entity ("Flagyl") in the question is also included in the answer. This undoubtedly makes the answer easier to extract as long as the model can recognize significant words and do "word matching".
To further explore how much clinical language understanding is needed and what kind of errors do the state-of-the-art reader make, we conduct error analysis using DocReader (Chen et al., 2017) (also used in (Pampari et al., 2018)) on the emrQA dataset. More specifically, we randomly sample 50 questions that are answered incorrectly by the model (based on exact match metric) from the Medication and Relation dev set respectively<sup>5</sup>. The results are shown in Table 3 (examples for each error type are also given for better understanding).
Since emrQA answers are often incomplete in the dataset, we deem span mismatch errors acceptable as long as the predictions include the key part of the ground truths. Surprisingly, span mismatch include key info errors, along with ambiguous questions, incorrect golds and false negatives (the prediction is correct but it is not in the emrQA answers) errors, which are caused by the dataset itself, account for  $90\%$  of total errors, suggesting that the accuracy of these models is even higher than we report.
Note that these 100 samples are sampled from errors, which are different from the previously sampled ones.
Another interesting finding from the error analysis is that to our surprise, only a very small amount (2%) of errors may have been caused by a lack of external domain knowledge while Pampari et al. (2018) claim that 39% of the questions in the emrQA dataset need domain knowledge. This surprising result might be due to: (1) neural models being able to encode relational or associative knowledge from the text corpora as has also been reported in recent studies (Petroni et al., 2019; Bouraoui et al., 2020), and (2) questions and answers sharing key phrases (as we mentioned earlier in §3.1) in many samples, making it more likely that fewer questions need external knowledge to be answered than previously reported.
4 Comprehensive Quantitative Analysis
In this section, we conduct comprehensive experiments on the emrQA dataset with state-of-the-art readers and recently dominating BERT models. Full experimental settings are described in Appendix A.
4.1 How redundant are the emrQA pairs?
Though there are more than 1 million questions in the emrQA dataset (as shown in Table 1), many questions and their patterns are very similar since they are generated from the same question templates. This observation leads to a natural question: do we really need so many questions to train an CliniRC system? If many questions are similar to each other, it is very likely that using a sampled subset can achieve roughly the same performance that is based on the entire dataset.
To verify our hypothesis, we first split the two datasets into train, dev, and test set with the proportion of 7:1:2 w.r.t. the contexts (full statistics are shown in Appendix Table A1). Then we randomly sample  $\{5\%, 10\%, 20\%, 40\%, 60\% \}$  and  $\{1\%, 3\%, 5\%, 10\%, 15\% \}$  of the QA pairs in each document (context) of the Medication and the Relation training sets respectively. We run DocReader (Chen et al., 2017) on the sampled subsets and evaluate them on the same dev and test set.
As shown in Figure 3, using  $20\%$  of the questions in the Medication and  $5\%$  of the questions in the Relation dataset can achieve roughly the same performance as using the entire training sets.
Figure 3: Impact of training size on the performance of DocReader (Chen et al., 2017) based on the Medication and Relation dataset.
These verify our hypothesis, and illustrate learning a good and robust reader system based on the emrQA dataset does not need so many question-answer pairs. While deep models are often data-hungry, it does not mean more data can always lead to better performance. In addition to the training size, diversity should also be considered as another important criterion for data quality.
In the following experiments, we use the sampled subsets (20% for Medication and 5% for Relation) considering the time and memory cost as well as performance.
4.2 Little room for improvement
Since the answers in emrQA are often incomplete, the performance of a model is more appropriately reflected by its F1 score. As shown in Table 2, we obtain F1 scores of  $74\%$  and  $95\%$  on two datasets respectively when we test human-labeled answers against the emrQA answers on a sampled dataset. We can see from Table 4 that the best performing reader, DocReader, achieves around  $70\%$  and  $94\%$  F1 performance on the Medication and Relation test set respectively, which are very close to the human performance just described. Though designing more complex and advanced models may achieve better scores, such scores are obtained w.r.t. noisy emrQA answers and may not translate meaningfully to real cases.
4.3 BERT does not always win
BERT models have achieved very promising results recently in various NLP tasks including RC (Devlin et al., 2019). We follow their experiment setting of BERT for doing reading comprehension on the SQuAD (Rajpurkar et al., 2016) dataset. To our surprise, as shown in Table 4, BERT models (BERT-base, its biomedical version BioBERT (Lee et al., 2019), and its clinical version ClinicalBERT
<sup>6</sup>The sampling percentage of the Relation dataset is smaller than the Medication dataset since the former one has more QA pairs (roughly 4 times).
4478
Model	Medication	Relation
Dev	Test	Dev	Test
EM	F1	EM	F1	EM	F1	EM	F1
BiDAF (Seo et al., 2017)	25.50	68.13	23.35	67.18	81.51	90.84	82.74	91.27
DocReader (Chen et al., 2017)	29.20	72.78	25.68	70.45	86.43	94.44	86.94	94.85
QANet (Yu et al., 2018)	27.67	69.40	24.74	67.34	82.41	90.61	82.68	91.56
BERT-base (Devlin et al., 2019)	26.62	68.75	24.00	67.49	80.17	90.01	83.29	92.38
BioBERT (Lee et al., 2019)	27.81	71.90	24.75	69.97	81.57	91.38	83.61	92.62
ClinicalBERT (Alsentzer et al., 2019)	27.14	71.84	24.06	69.05	83.12	91.96	85.33	93.06
Table 4: Overall performance of all models on the Medication and Relation dataset. All numbers are percentages.
(Alsentzer et al., 2019)) do not dominate as they do in the open-domain RC tasks. The reasons may be three-fold: 1) BERT benefits the most from large training corpora. The training corpora of BERT-base and BioBERT are Wikipedia + BookCorpus (Zhu et al., 2015) and PubMed articles respectively, both of which may have different vocabularies and use different language expressions from clinical texts. Though ClinicalBERT was pretrained on MIMIC-III (Johnson et al., 2016) clinical texts, the training size of the corpus ( $\sim$ 50M words) is far less than that used in BERT ( $\sim$ 3300M words), which may make the model less powerful as it is on the open-domain tasks. 2) Longer Contexts. As can be seen from Table 1, the number of tokens in the contexts is commonly larger than open-domain RC datasets like SQuAD ( $\sim$ 1000 v.s. $\sim$ 116 avg). We suspect that long contexts might make it more challenging to model sequential information. For sequences that are longer than the max length of the BERT model, they are truncated into a set of short sequences, which may hinder the model from capturing long dependencies (Dai et al., 2019) and global information in the entire document. 3) Easy Questions. Another possible reason might be the question patterns are too easy and a simpler reader with far less parameters can learn the patterns and obtain satisfying performance.
Additionally, to further evaluate the models in the fine-grained level, inspired by (Gururangan et al., 2018), we partition the Medication and Relation test sets into Easy and Hard subsets using a base model. The details of Easy/Hard splits can be found in Appendix C. As can be seen from Table A4, most of the questions in the two datasets are easy, which indicates the emrQA dataset might not be challenging for the current QA models. More difficult datasets are needed to advance the Clinical Reading Comprehension task.
5 Desiderata in Real-World CliniRC
Following our analysis of the emrQA dataset, we further study two aspects of clinical reading comprehension systems that we believe are crucial for their real-world applicability: the need to encode clinical domain knowledge and to generalize to unseen questions and documents.