SlowGuess/ACL_2020_OCR huggingface dataset:
Yuanzhe Zhang $^{1}$ , Zhongtao Jiang $^{1,2}$ , Tao Zhang $^{1,2}$ , Shiwan Liu $^{1,2}$ , Jiarun Cao $^{3*}$ , Kang Liu $^{1,2}$ , Shengping Liu $^{4}$  and Jun Zhao $^{1,2}$
$^{2}$  School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100049, China
$^{3}$  National Centre for Text Mining, University of Manchester, Manchester, M1 7DN, United Kingdom
$^{4}$  Beijing Unisound Information Technology Co., Ltd, Beijing, 100028, China {yzzhang, zhongtao.jiang, Tao.zhang, shiwan.liu, kliu, jzhao} @nlpr.ia.ac.cn jiarun.cao@manchester.ac.uk, liushengping@unisound.com
Electronic Medical Records (EMRs) have become key components of modern medical care systems. Despite the merits of EMRs, many doctors suffer from writing them, which is time-consuming and tedious. We believe that automatically converting medical dialogues to EMRs can greatly reduce the burdens of doctors, and extracting information from medical dialogues is an essential step. To this end, we annotate online medical consultation dialogues in a window-sliding style, which is much easier than the sequential labeling annotation. We then propose a Medical Information Extractor (MIE) towards medical dialogues. MIE is able to extract mentioned symptoms, surgeries, tests, other information and their corresponding status. To tackle the particular challenges of the task, MIE uses a deep matching architecture, taking dialogue turn-interaction into account. The experimental results demonstrate MIE is a promising solution to extract medical information from doctor-patient dialogues.<sup>1</sup>
With the advancement of the informatization process of the medical system, Electronic Medical Records (EMRs) are required by an increasing number of hospitals all around the world. Compared with conventional medical records, EMRs are easy to save and retrieve, which bring considerable convenience for both patients and doctors. Furthermore, EMRs allow medical researchers to investigate the implicit contents included, such as epidemiologic study and patient cohorts finding.
Despite the advantages, most doctors complain that writing EMRs makes them exhausted (Wachter and Goldsmith, 2018). According to the study of Sinsky et al. (2016), physicians spend nearly two hours doing administrative work for every hour of face-time with patients, and the most time-consuming aspect is inputting EMRs.
We believe that automatically converting doctor-patient dialogues into EMRs can effectively remove the heavy burdens of doctors, making them more deliberate to communicate with their patients. One straightforward approach is the end-to-end learning, where more supervised data, i.e., dialogue-EMR pairs are needed. Unfortunately, such data is hard to acquire in medical domain due to the privacy policy. In this paper, We focus on extracting medical information from dialogues, which we think is an essential step for EMR generation.
Extracting information from medical dialogues is an emerging research field, and there are only few previous attempts. Finley et al. (2018) proposed an approach that consists of five stages to convert a clinical conversation to EMRs, but they do not describe the detail method. Du et al. (2019) also focused on extracting information from medical dialogues, and successfully defined a new task of extracting 186 symptoms and their corresponding status. The symptoms were relatively comprehensive, but they did not concern other key information like surgeries or tests. Lin et al. (2019) collected online medical dialogues to perform symptom recognition and symptom inference, i.e., inference the status of the recognized symptoms. They also used the sequential labeling method, incorporated global attention and introduced a static symptom graph.
There are two main distinctive challenges for tackling doctor-patient dialogues: a) Oral expres
*Contribution during internship at Institute of Automation, Chinese Academy of Sciences.
<sup>1</sup>Data and codes are available at https://github.com/nlpir2020/MIE-ACL-2020.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6460-6469 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Dialogue Window	Annotated Labels
Patient: Doctor, could you please tell me is it premature beat?Doctor: Yes, considering your Electrocardiogram. Do youfeel palpitation or short of breath?Patient: No. Can I do radiofrequency ablation?Doctor: It is worth considering. Any discomfort in chest?Patient: I always have bouts of pain.	Symptom: Premature beat (doctor-pos)Test: Electrocardiogram (patient-pos)Symptom: Cardiopalmus (patient-neg)Symptom: Dyspnea (patient-neg)Surgery: Radiofrequency ablation(doctor-pos)Symptom: Chest pain (patient-pos)
Figure 1: A typical medical dialogue window and the corresponding annotated labels. "Pos" is short for "positive" and "neg" is short for "negative". Text color and label color are aligned for clarity. All the examples in the paper are translated from Chinese.
sions are much more diverse than general texts. There are many medical terms in the dialogue, but many of them are not uttered formally, which will lead to performance degradation of conventional Natural Language Processing (NLP) tools. b) Available information is scattered in various dialogue turns, thus the interaction between turns should be also considered. In order to meet these challenges, we first annotate the dialogues in a window-sliding style, as illustrated in Figure 1. Then, we propose MIE, a Medical Information Extractor constructed on a deep matching model. We believe our annotation method could put up with informal expressions, and the proposed neural matching model is able to harness the turn-interactions.
We collect doctor-patient dialogues from a popular Chinese online medical consultation website, Chunyu-Doctor  $^{2}$ , where medical dialogues are in text format. We focus on the cardiology domain, because there are more inquiries and less tests than other departments. The annotation method considers both effectiveness and feasibility. We define four main categories, including symptoms, tests, surgeries and other information, and we further define frequent items in the categories and their corresponding status at the same time. There are two merits of our annotation method: a) the annotation is much easier than the sequential labeling manner and does not need the labelers to be medical experts; b) we can annotate the circumstances that a single label is expressed by multiple turns. We totally annotate 1,120 dialogues with 18,212
segmented windows and obtain more than 40k labels.
We then develop MIE constructed on a novel neural matching model. MIE model consists of four main components, namely encoder module, matching module, aggregate module and scorer module. We conduct extensive experiments, and MIE achieves a overall F-score of 69.28, which indicates our proposed approach is a promising solution for the task.
To sum up, the contributions of this paper are as follows:
- We propose a new dataset, annotating 1,120 doctor-patient dialogues from online consultation medical dialogues with more than 40k labels. The dataset will help the following researchers.
- We propose MIE, a medical information extractor based on a novel deep matching model that can make use of the interaction between dialogue turns.
- MIE achieves a promising overall F-score of 69.28, significantly surpassing several competitive baselines.
Extracting information from medical texts is a long-term objective for both biomedical and NLP community. For example, The 2010 i2b2 challenge provides a popular dataset still used in many recent researches (Uzuner et al., 2011). Three tasks were presented: a concept extraction task focused on the
$^{2}$ https://www.chunyuyisheng.com
6461
extraction of medical concepts from patient reports; an assertion classification task focused on assigning assertion types for medical problem concepts; a relation classification task focused on assigning relation types that hold between medical problems, tests, and treatments.
Extracting medical information from dialogues just gets started. Finley et al. (2018) proposed a pipeline method to generate EMRs. The approach contains five steps: dialogue role labeling, Automatic Speech Recognition (ASR), knowledge extraction, structured data processing and Natural Language Generation (NLG) (Murty and Kabadi, 1987). The most important part is knowledge extraction, which uses dictionary, regular expression and other supervised machine learning methods. However, the detailed explanations are left out, which make us hard to compare with them.
Du et al. (2019) aimed at generating EMRs by extracting symptoms and their status. They defined 186 symptoms and three status, i.e., experienced, not experienced and other. They proposed two models to tackle the problem. Span-Attribute Tagging Model first predicted the span of a symptom, and then used the context features to further predict the symptom name and status. The seq2seq model took  $k$  dialogue turns as input, and then directly generated the symptom name and status. They collected incredible 90k dialogues and annotated 3k of them, but the dataset is not public.
The most similar work to ours is (Lin et al., 2019), which also annotated Chinese online medical dialogues. Concretely, they annotated 2,067 dialogues with the BIO (begin-in-out) schema. There are two main components, namely symptom recognition and symptom inference in their approach. The former utilized both document-level and corpus-level attention enhanced Conditional Random Field (CRF) to acquire symptoms. The letter serves determining the symptom status.
Our work differs from (Du et al., 2019) and (Lin et al., 2019) mainly in the following two points: a) we only extract 45 symptom items, but the status are more detailed, furthermore, we extract surgeries, tests and other information; b) we use different extracting method. Since the annotation system is different, our approach does not need the sequential labeling, which relieves the labeling work.
3 Corpus Description
3.1 Annotation Method
We collect doctor-patient dialogues from a Chinese medical consultation website, Chunyu-Doctor. The dialogues are already in text format. We select cardiology topic consultations, since there are more inquiries, while dialogues of other topics often depend more on tests. A typical consultation dialogue is illustrated in Figure 1. The principle of the annotation is to label useful information as comprehensive as possible.
A commonly utilized annotation paradigm is sequential labeling, where the medical entities are labeled using BIO tags (Du et al., 2019; Lin et al., 2019; Collobert et al., 2011; Huang et al., 2015; Ma and Hovy, 2016). However, such annotation methods cannot label information that a) expressed by multiple turns and b) not explicitly or not consecutively expressed. Such situations are not rare in spoken dialogues, as can be seen in Figure 1.
To this end, we use a window-to-information annotation method instead of sequential labeling. As listed in Table 1, we define four main categories, and for each category, we further define frequent items. The item quantity of symptom, surgery, test and other info is 45, 4, 16 and 6, respectively. In medical dialogues, status is quite
Category	Item	Status
Symptom	Backache	patient-positive (appear)
Perspiration	patient-negative (absent)
Hiccups	doctor-positive (diagnosed)
Nausea	doctor-negative (exclude)
Cyanosis	unknown
Fever
Fatigue
Abdominal discomfort
...
Surgery	Interventional treatment	patient-positive (done)
Radiofrequency ablation	patient-negative (not done)
Heart bypass surgery	doctor-positive(suggest)
Stent implantation	doctor-negative (deprecated)
unknown
Test	B-mode ultrasonography
CT examination	patient-positive(done)
CT angiography	patient-negative (not done)
CDFI	doctor-positive(suggest)
Blood pressure measure-ment	doctor-negative (deprecated)
Ultrasonography	unknown
MRI
Thyroid function test
Treadmill test
...
Other info	Sleep	patient-positive (normal)
Diet	patient-negative (abnormal)
Mental condition	unknown
Defecation
Smoking
Drinking
Table 1: The detailed annotation labels of the dataset.
6462
crucial that cannot be ignored. For example, for a symptom, the status of appearance or absence is opposite for a particular diagnose. So it is necessary to carefully define status for each category. The status options vary with different categories, but we use unified labels for clarity. The exact meanings of the labels are also explained in Table 1.
The goal of annotation is to label all the predefined information mentioned in the current dialogue. As the dialogues turn to be too long, it is difficult for giving accurate labels when finishing reading them. Thus, we divide the dialogues into pieces using a sliding window. A window consists of multiple consecutive turns of the dialogue.
It is worth noting that the window-sliding annotations can be converted into dialogue-based ones like dialogue state tracking task (Mrkšić et al., 2017), the later annotation state will overwrite the old one. Here, the sliding window size is set to 5 as Du et al. (2019) did, because this size allows the included dialogue turns contain proper amount of information. For windows with less than 5 utterances, we pad them at the beginning with empty strings. The sliding step is set to 1.
We invite three graduate students to label the dialogue windows. The annotators are guided by two physicians to ensure correctness. The segmented windows are randomly assigned to the annotators.
In all, we annotate 1,120 dialogues, leading to 18,212 windows. We divide the data into train/develop/test sets of size 800/160/160 for dialogues and 12,931/2,587/2,694 for windows, respectively. In total, 46,151 labels are annotated, averaging 2.53 labels in each window, 41.21 labels in each dialogue. Note that about  $12.83\%$  of windows have no gold labels, i.e., there is no pre-defined information in those windows. The distribution of the labels is shown in Table 2. The status distribution is shown in Table 3. The annotation consistency, i.e., the cohen's kappa coefficient (Fleiss and Cohen, 1973) of the labeled data is 0.91, which means our annotation approach is feasible and easy to follow.
Dialogue	Window	Symptom	Surgery	Test	Other info
Train	800	12931	21420	839	8879	1363
Dev	160	2587	4254	119	1680	259
Test	160	2694	4878	264	1869	327
Total	1120	18212	30552	1222	12428	1949
Table 2: The detailed annotation statistics of the dataset.
Patient-pos	Patient-neg	Doctor-pos	Doctor-neg	Unknown
Symptom	15119	1782	1655	910	11086
Surgery	169	48	698	10	297
Test	5589	303	4443	44	2049
Other info	550	1399	-	-	1505
Table 3: The distribution of status over all labels.
We evaluate the extracted medical information results as ordinary information extraction task does, i.e., Precision, Recall and F-measure. To further discover the model behavior, we set up three evaluation metrics from easy to hard. Category performance is the most tolerant metric. It merely considers the correctness of the category. Item performance examines the correctness of both category and item, regardless of status. Full performance is the most strict metric, meaning that category, item and the corresponding status must be completely correct.
We will report both window-level and dialogue-level results.
Window-level: We evaluate the results of each segmented window, and report the micro-average of all the test windows. Some windows have no gold labels, if the prediction on a window with no gold labels is also empty, it means the model performs well, so we set the Precision, Recall and F-measure to 1, otherwise 0.
Dialogue-level: First we merge the results of the windows that belong to the same dialogue. For labels that are mutually exclusive, we update the old labels with the latest ones. Then we evaluate the results of each dialogue, and finally report the micro-average of all the test dialogues.
In this section, we will elaborate the proposed MIE model, a novel deep matching neural network model. Deep matching models are widely used in multiple natural language processing tasks such as machine reading comprehension (Seo et al., 2017; Yu et al.), question answering (Yang et al., 2016) and dialogue generation (Zhou et al., 2018; Wu et al., 2017). Compared with classification models, matching models are able to introduce more information of the candidate side and promote interaction between both ends.
The architecture of MIE is shown in Figure 2. There are four main components, namely encoder module, matching module, aggregate module and scorer module. The input of MIE is a doctor-patient
6463
Figure 2: The architecture of MIE model.
dialogue window, and the output is the predicted medical information.
Encoder Module
The encoder is implemented by Bi-LSTM (Hochreiter and Schmidhuber, 1997) with self-attention (Vaswani et al., 2017). Let the input utterance be  $X = (x_{1}, x_{2}, \dots, x_{l})$ , the encoder works as follows:
$$
H = \operatorname {B i L S T M} (X)
$$
$$
a [ j ] = W H [ j ] + b
$$
$$
p = \operatorname {s o f t m a x} (a) \tag {1}
$$
$$
c = \sum_ {j} p [ j ] H [ j ]
$$
We denote  $H, c = \operatorname{Encoder}(X)$  for brevity.  $H$  consists of contextual representations of every token in input sequence  $X$ , and  $c$  is a single vector that compresses the information of the entire sequence in a weighted way.
We denote a window with  $n$  utterances as  $\{U[1],\ldots U[n]\}$ . For a candidate consists of category, item and status like Symptom:Heart failure (patient-positive), we split it to category-item pair Symptom:Heart failure denoted by  $V$  and status patient-positive denoted by  $S$ . To introduce more oral information, we also add item-related colloquial expressions collected during the annotation to the end of  $V$ . Having defined the basic structure of the encoder, we now build representations for utterances  $U$  in the dialogue window, and the candidate category-item
pair  $V$  and its status  $S$ :
$$
H _ {c} ^ {u t t} [ i ], c _ {c} ^ {u t t} [ i ] = \operatorname {E n c o d e r} _ {c} ^ {u t t} (U [ i ])
$$
$$
H _ {s} ^ {u t t} [ i ], c _ {s} ^ {u t t} [ i ] = \operatorname {E n c o d e r} _ {s} ^ {u t t} (U [ i ]) \tag {2}
$$
$$
H _ {c} ^ {\text {c a n}}, c _ {c} ^ {\text {c a n}} = \operatorname {E n c o d e r} _ {c} ^ {\text {c a n}} (V)
$$
$$
H _ {s} ^ {c a n}, c _ {s} ^ {c a n} = \operatorname {E n c o d e r} _ {s} ^ {c a n} (S)
$$
Where the superscript  $utt$  and can represents utterance encoder and candidate encoder respectively, the subscript  $c$  and  $s$  represents category encoder and status encoder respectively, and  $i \in [1, n]$  is the index of utterance in the dialogue window. All the candidates will be encoded in this step, but we only illustrate one in the figure and equations for brevity. Note that  $U, V, S$  is encoded with encoders differ from utterance to candidate and from category to status in order to make each encoder concentrate on one specific type (category-specific and status-specific) of information.
Matching Module
In this step, the category-item representation is treated as a query in attention mechanism to calculate the attention values towards original utterances. Then we can obtain the category-specific representation of utterance  $U[i]$  as  $q_{c}[i]$ .
$$
a _ {c} [ i, j ] = c _ {c} ^ {c a n} \cdot H _ {c} ^ {u t t} [ i, j ]
$$
$$
p _ {c} [ i ] = \operatorname {s o f t m a x} \left(a _ {c} [ i ]\right) \tag {3}
$$
$$
q _ {c} [ i ] = \sum_ {j} p _ {c} [ i, j ] H _ {c} ^ {u t t} [ i, j ]
$$
Meanwhile, the status representation is treated as another query to calculate the attention values
towards original utterances. Then we can obtain the status-specific representation of utterance  $U[i]$  as  $q_{s}[i]$ .
$$
a _ {s} [ i, j ] = c _ {s} ^ {\text {c a n}} \cdot H _ {s} ^ {\text {u t t}} [ i, j ]
$$
$$
p _ {s} [ i ] = \operatorname {s o f t m a x} \left(a _ {s} [ i ]\right) \tag {4}
$$
$$
q _ {s} [ i ] = \sum_ {j} p _ {s} [ i, j ] H _ {s} ^ {u t t} [ i, j ]
$$
Where  $[i,j]$  denotes the  $j$ th word in the  $i$ th utterance. The goal of this step is to capture the most relevant information from each utterance given a candidate. For example, if the category-item pair of the candidate is Symptom: Heart failure, the model will assign high attention values to the mentions of heart failure in utterances. If the status of the candidate is patient-positive, the attention values of expressions like "I have", "I've been diagnosed" will be high. So the matching module is important to determine the existence of a category-item pair and status related expressions.
Aggregate Module
The matching module introduced above have captured the information of the existence of category-item pairs and status. To know whether a candidate is expressed in a dialogue window, we need to obtain the category-item pair information and its status information together. In particular, we need to match every category-item representation  $q_{c}[i]$  with  $q_{s}[i]$ .
Sometimes the category-item pair information and its status information appear in the same utterance. But sometimes, they will appear in different utterances. For example, many question-answer pairs are adjacent utterances. So we need take the interactions between utterances into account. Based on this intuition, we define two kinds of strategies to get two different models.
MIE-single: The first strategy assumes that the category-item pair information and its status information appear in the same utterance. The representation of the candidate in the  $i$ th utterance is a simple concatenation of  $q_{c}[i]$  and  $q_{s}[i]$ :
$$
f [ i ] = \operatorname {c o n c a t} \left(q _ {c} [ i ], q _ {s} [ i ]\right) \tag {5}
$$
Where  $f[i]$  consists of category-item pair and its status which can be used to predict the score of the related candidate. The model only considers the interaction within a single utterance.
The acquired representations are independent from each other. This model is called MIE-single.
MIE-multi: The second strategy considers the interaction between the utterances. To obtain the related status information of other utterances, we treat  $q_{c}[i]$  as a query to get the attention values towards the representations of status, i.e.,  $q_{s}$ . Then we can obtain the candidate representation of the utterance:
$$
a [ i, k ] = q _ {c} [ i ] ^ {T} W q _ {s} [ k ]
$$
$$
p [ i ] = \operatorname {s o f t m a x} (a [ i ])
$$
$$
\widetilde {q} _ {s} [ i ] = \sum_ {k} p [ i, k ] q _ {s} [ k ] \tag {6}
$$
$$
f [ i ] = \operatorname {c o n c a t} \left(q _ {c} [ i ], \widetilde {q} _ {s} [ i ]\right)
$$
Where  $W$  is a learned parameter, and  $\widetilde{q}_s$  is the new representation of the status, containing the relative information of other utterances. The utterance order is an important clue in a dialogue window. For example, the category-item pair information can hardly related to status information whose utterance is too far. In order to capture this kind of information, we also take utterance position into account. Concretely, we add positional encoding (Vaswani et al., 2017) to each  $q_c$  and  $q_s$  at the beginning. We denote this model as MIE-multi.
The output of the aggregate module contains the information of a entire candidate, including category-item and status information.
Scorer Module
The output of the aggregate module is fed into a scorer module. We use each utterance's feature  $f[i]$  to score the candidate, as it is already the candidate-specific representation. The highest score of all the utterances in the window is the candidate's final score:
$$
s ^ {u t t} [ i ] = \operatorname {f e e d f o r w a r d} (f [ i ]) \tag {7}
$$
$$
y = \operatorname {s i g m o i d} \left(\max  \left(s ^ {u t t} [ i ]\right)\right)
$$
Where feedforward is a 4 layer full-connection neural network.
The loss function is the cross entropy loss defined as follows:
$$
L = \frac {1}{K L} \sum_ {k} \sum_ {l} - y _ {l} ^ {k} \log \left(\widehat {y} _ {l} ^ {k}\right) + \tag {8}
$$
$$
\left(1 - y _ {l} ^ {k}\right) \log \left(1 - \widehat {y} _ {l} ^ {k}\right)
$$