SlowGuess/ACL_2020_OCR huggingface dataset:
Baidu Inc, Beijing, China
{chenjun22,daixiaoya,yuanquan02,luchao,huanghaifeng}@baidu.com
The automatic text-based diagnosis remains a challenging task for clinical use because it requires appropriate balance between accuracy and interpretability. In this paper, we attempt to propose a solution by introducing a novel framework that stacks Bayesian Network Ensembles on top of Entity-Aware Convolutional Neural Networks (CNN) towards building an accurate yet interpretable diagnosis system. The proposed framework takes advantage of the high accuracy and generality of deep neural networks as well as the interpretability of Bayesian Networks, which is critical for AI-empowered healthcare. The evaluation conducted on the real Electronic Medical Record (EMR) documents from hospitals and annotated by professional doctors proves that, the proposed framework outperforms the previous automatic diagnosis methods in accuracy performance and the diagnosis explanation of the framework is reasonable.
The automatic diagnosis of diseases has drawn the increasing attention from both research communities and industrial companies in the recent years due to the advancement of artificial intelligence (AI) (Liang et al., 2019; Esteva et al., 2019; Liu et al., 2018). As reported in (Anandan et al., 2019), "AI-enabled analysis software is helping to guide doctors and other health-care workers through diagnostic processes and questioning to arrive at treatment decisions with greater speed and accuracy." Although the image-based diagnosis has been well studied using PACS (Picture Archiving and Communication Systems) data (Litjens et al., 2017), the text-based diagnosis for Clinical Decision Support (CDS) (Berner, 2007) remains difficult due to the rare access to reliable clinical corpus and the difficulty in balancing between accuracy and interpretability.
Table 1: A real outpatient EMR from hospital.
Section	Content
Basic	男,30岁(Male,30 years old)
CC	咽部不适3天(Pharyngeal discomfort for 3 days)
HPI	患者于3日前起咽痛伴发热,无呼吸困难、咳嗽、咳痰、嗳气或反酸(The patient developed pharyngalgia and fever 3 days ago, without dyspnea, cough, sputum, belching or acid reflux)
PE	咽峡稍充血,双侧扁桃体I度肿大,无栓塞物及瘢痕(The hypopharyngeal isthmus is slightly congested. The bilateral tonsils are first-degree enlarged. There is no embolism or scar in the pharynx.)
TR	血常规示白细胞计数升高,WBC12.5*10^9/L.C反应蛋白正常.(The blood test showed elevated white blood cell count,WBC12.5*10^9/L.The C-reactive protein is normal.)
Diagnosis	急性扁桃体炎(Acute tonsillitis)
There have been attempts to study automatic text-based diagnosis with Electronic Medical Record (EMR) documents integrated in the Hospital Information System (Mullenbach et al., 2018; Yang et al., 2018; Girardi et al., 2018). Basically, an EMR document is written by a doctor and consists of several sections that describe the illness of the patient. Besides the patient's basic information like name, age and gender, an EMR document contains Chief Complaint (CC), History of Present Illness (HPI), Physical Examination (PE), Test Reports (TR, e.g. lab test reports and PACS reports), Diagnosis, etc. Table 1 shows a real outpatient EMR document from a hospital. These sections describe the patient's medical situation from different aspects: CC summarizes the patient's main discomforts of this visit. HPI extends CC by adding more details and findings from the conversation between doctor and patient. PE shows the findings by physically examining the patient's body, e.g. by palpation or inspection. TR are the objective findings from the lab test reports or the PACS reports. In the hospitals, the doctors will make a comprehensive analysis mainly based on CC, HPI, PE, TR and the basic information, and make a diagnosis. However, it is very hard for computers to automatically understand all the diverse sections and capture the key
3143
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3143-3153 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
information before making an appropriate diagnosis. Besides, an inpatient EMR document is similar to that in Table 1 except that HPI, PE and TR are usually more lengthy and detailed. The framework proposed in this work can be applied on both the outpatient and the inpatient EMR documents and we will not distinguish them later.
In this study, we bring forward a novel framework of automatic diagnosis with EMR documents for CDS. Specifically, we propose to predict the main diagnosis based on the patient's current illness. Different from the previous works (Yang et al., 2018; Sha and Wang, 2017; Li et al., 2017; Girardi et al., 2018; Mullenbach et al., 2018) that solely rely on the end-to-end neural models, we propose to stack the Bayesian Network (BN) ensembles on top of Entity-aware Convolutional Neural Networks (ECNN) in automatic diagnosis, where ECNN improves the accuracy of the prediction and BN ensembles explain the prediction. The proposed framework attempts to bring some interpretability of the predictions by incorporating the knowledge encoded in the BN ensembles. The main contributions of this work are as follows:
- We propose a novel framework that stacks the Bayesian network ensembles on top of the entity-aware convolutional neural networks to bring interpretability into automatic diagnosis without compromising the accuracy of deep learning. Interpretability is very important in the AI-empowered healthcare studies.
- We bring forward three variants of Bayesian Networks for disease inference that provides interpretability. Moreover, we ensemble these BNs towards more robust diagnosis results.
- The evaluation conducted on real EMR documents from hospitals proves that the proposed framework outperforms the previous automatic diagnosis methods with EMRs. The proposed framework has been used as a critical component in the clinical decision support system developed by Baidu, which assists physicians in diagnosis in over hundreds of primary healthcare facilities in China.
- We publish the Chinese medical knowledge graph of Gynaecology and Respiration used in our Bayesian Network for disease inference with this paper for reproducibility. The data
set can be downloaded from Github.2
Due to the rapid advancement of machine intelligence, the text-based automatic diagnosis is becoming one of the most important applications of machine learning and natural language processing in the recent years (Anandan et al., 2019; Koleck et al., 2019). Different from diagnosis or question answering on the Web (Chen et al., 2019), diagnosis for the CDS takes place in the hospitals and clinics, and the predictive algorithm is integrated into the Hospital Information System to assist doctors and physicians in the diagnosis.
Liang et al. (2019) proposes a top-down hierarchical classification method towards diagnosing pediatric diseases. From the root to the leaf, each level on the diagnostic hierarchy is a logistic regression model that performs classification on labels from coarse granularity to fine-grained granularity, e.g. from organ systems down to respiratory systems and to upper respiratory systems. This method requires heavy manual annotation of training samples at different levels of hierarchy.
Zhang et al. (2017) combines the variational auto-encoder and the variational recurrent neural network together to make diagnosis based on laboratory test data. However, laboratory test data are not the only resources considered in this paper.
Prakash et al. (2017) introduces the memory networks into diagnostic inference based on free text clinical records with external knowledge source from Wikipedia.
Sha and Wang (2017) proposes a hierarchical GRU-based neural network to predict the clinical outcomes based on the medical code sequences of the patient's previous visits. It deals with the sequential disease forecasting problem with EHR data rather than the diagnosis problem for the current visit with EMR document. Similarly, Choi et al. (2016a) studies the RNN-based model for clinical event prediction. Baumel et al. (2017) investigates the multi-label classification problem for discharge summaries of EHR with hierarchical attention-bidirectional GRU.
The most similar works to ours are in (Yang et al., 2018; Li et al., 2017) which trains an end-to-end convolutional network model to predict di
Different from Electronic Health Record (EHR) where the illness of a patient's multiple visits are combined together, EMR only contains the patient's illness of this particular visit. EMRs are more generally used in the hospitals in China.
$^{2}$ https://github.com/PaddlePaddle/ Research/tree/master/KG/ACL2020_SignOrSymptom_Relationship
agnosis based on EMRs. Besides, Girardi et al. (2018) improves the CNN model with the attention mechanism in automatic diagnosis. Moreover, Mullenbach et al. (2018) studies a label-wise attention model to further improve the accuracy of diagnosis at the cost of more computation time. Choi et al. (2016b) proposes a reverse time attention mechanism for interpretable healthcare studies.
Different from the previous studies, the novelty of this paper is to bring interpretability into automatic diagnosis by stacking the ensembles of Bayesian networks on top of the entity-aware convolutional neural networks.
Automatic diagnosis can be formally considered as a classification problem where the proposed method outputs a probability distribution  $\operatorname{Pr}(d|\mathbf{S})$  over all diseases  $d\in \mathbf{D}$  based on the illness description  $\mathbf{S}$ . In this study,  $\mathbf{S}$  corresponds to the patient's EMR document, i.e.  $\mathbf{S}$  consists of several sections of texts and some structured data like age, gender and medical department.
We bring forward a new framework that combines the black-box deep learning and the white-box knowledge inference to diagnose disease with EMR documents. Figure 1 shows the architecture of the proposed framework. Firstly, the medical entities are extracted from the EMR contents. Then, the EMR document is fed into the entity-aware convolutional networks to generate disease prior probability. Next, the Bayesian network ensembles perform disease inference based on the prior probability and the probabilistic graphical models (PGMs) before ensembling the final predictions.
3.1 Named Entity Recognition
Before introducing the convolutional and the Bayesian networks, we first discuss a basic component of this framework – the named entity recognition (NER). NER extracts the entities as well as their types from text sentences, which is very important to capture the key information of the texts. In our experiments, we used Baidu's enterprise Chinese medical NER system that integrates the advanced NER models (Dai et al., 2019; Jia et al., 2019) and extracts entities of symptoms, vital signs, diseases and test report findings.
The F1 score of the NER system we use is  $91\%$  in a separate evaluation conducted on 1000 deduplicated sentences from real EMR documents by 10
Table 2: The NER results of the EMR document shown in Table 1. TR Finding: test result finding. (+) for positive, (-) for negative and (?) for unknown.
Word	Section	Type	Polarity
咽部不适(pharyngeal discomfort)	CC	Symptom	(+)
咽痛(pharyngalgia)	HPI	Symptom	(+)
发热(fever)	HPI	Symptom	(+)
呼吸困难(dyspnea)	HPI	Symptom	(-)
咳嗽(cough)	HPI	Symptom	(-)
咳痰(sputum)	HPI	Symptom	(-)
嗳气 belching)	HPI	Symptom	(-)
反酸acid reflux)	HPI	Symptom	(-)
咽峡充血 (congested hypopharyngeal isthmus)	PE	Vital Sign	(+)
双侧扁桃体肿大(enlarged bilateral tonsils)	PE	Vital Sign	(+)
咽部栓塞物(pharyngeal embolism)	PE	Vital Sign	(-)
咽部瘢痕(pharyngeal scar)	PE	Vital Sign	(-)
白细胞计数升高(elevated WBC)	TR	TR Finding	(+)
C反应蛋白异常(abnormal C-reactive protein)	TR	TR Finding	(-)
急性扁桃体炎(acute tonsillitis)	Diagnosis	Disease	(+)
certificated physicians in China.  ${}^{3}$  Meanwhile,the polarity (positive (+),negative (-) or unknown (?) of entities is also recognized. The polarity in this work objectively means the presence or absence of a finding in a given EMR. It is recognized in conjunction with the rule-based method with a vocabulary of negative Chinese words as well as the polarity detection model. Table 2 shows the NER results of the EMR in Table 1. Please note that the disease (acute tonsillitis) from the diagnosis section is the ground-truth label to predict and it will not be included in the input to the predictive model in the evaluation.
In the offline processing of the EMR corpus, we preserved the Top-  $K$  most frequent entities of all types as the entity vocabulary. In later experiments, we empirically set  $K = 10,000$ . The entity vocabulary will be used to construct the one-hot feature for each EMR document, which will be introduced later. Since NER is not the focus of this study, the readers can choose the public Chinese NER API from Baidu for fast experiments. We will focus on the major contributions of the proposed framework in the next sections.
3There are two senior physicians beyond the attending doctor level and eight junior physicians contributed in the annotation tasks here and later.
<sup>4</sup>http://ai.baidu.com/tech/cognitive-entity_annotation
Figure 1: The architecture of the proposed framework.
3.2 ECNN for Prior Generation
The convolutional networks take as input the list of texts w.r.t. the sections of an EMR document as well as the medical entities extracted from them, and output the probability distribution of the diseases. To distinguish from the previous CNN models without medical entities (Yang et al., 2018; Li et al., 2017), we use ECNN to denote the entity-aware CNN model proposed in this paper where another branch of fully connected layers processes the medical entities and outputs the corresponding feature representation. Let  $N$  denote the number of sections (CC, HPI, PE, TR, etc) selected from the EMR document to construct ECNN. ECNN consists of two parts: (1)  $N$  convolutional towers, each of which reads a unique section, and (2) one multi-layer perceptron (MLP) branch that reads a high-dimensional hand-crafted feature.
Similar to the previous CNN method for text classification (Kim, 2014), each convolutional tower processes the input sequence with three kernels of various length resulting in multi-channel feature output. The three kernels process the input with 3-grams, 4-grams and 5-grams, respectively, and their outputs are concatenated as the output of a convolutional tower. Each kernel in the convolutional networks has 100 filters with strides as 1. The input is padded with valid method and the output is activated by ReLU.
For the input of MLP, we create the entity vocabulary that consists of the top-  $K$  frequent entities. Then, each EMR document is transformed to a  $K$ -dimensional one-hot feature  $f$ . That is, if the  $i$ -th entity in the entity vocabulary appears as a positive finding in the input EMR, then the  $i$ -th dimension
of  $f$  is set to 1, and otherwise, it is set to 0. Moreover, the patient's age and gender are appended to  $f$  to get the hand-crafted feature for MLP. The MLP contains one dense layer activated by sigmoid function with 128 hidden units.
ECNN is trained with Adam optimizer (learning rate 0.001), 20 epochs and batch size of 32. The output of each convolutional tower and the output of the MLP are further concatenated before passing through the dropout and the softmax layer. Similar to Kim (2014), the dropout rate is empirically set to 0.5. A  $|\mathbf{D}|$ -dimensional feature is output by ECNN as the disease priors for the inference in the next where  $\mathbf{D}$  is the disease set.
In ECNN, the CNNs are supposed to capture the sequential signals in the section texts and the MLP is supposed to encode the feature of the critical entities. By jointly modeling with CNNs and MLP, the proposed ECNN is expected to have superior performance than either of them alone.
3.3 Bayesian Network Ensembles
Although ECNN also outputs a probability distribution over all diseases, the result is not interpretable due to its end-to-end nature. However, the interpretability is very important in the CDS to explain how the diagnosis is generated by machines. Thus, we propose the Bayesian network ensembles on top of the output of ECNN to explicitly infer disease with PGMs. There are three steps:
3.3.1 Relation Extraction
We extract the relations between disease and other types of entities (disease, finding) where finding can be symptom, vital sign, test report finding, etc.
The rest of this paper will use finding to denote any type of entities other than disease. Relation extraction is performed in conjunction with the (disease, finding) co-occurrence mining and the deep extraction model (Shi et al., 2019) from the EMR documents and the textbooks  $^{5}$ . Then, the pairs with high co-occurrences larger than a support (e.g. 5) are preserved. The extracted relations are reviewed by 10 certificated physicians. The invalid extracted relations which result from issues like incorrect recognition of entities or polarities by NER, the symptom caused by the secondary diagnosis but incorrectly paired with the first diagnosis, are removed before adding to the medical knowledge graph. Therefore, the relation (disease, finding) in the medical knowledge graph can, to some extent, be interpreted as: disease causes finding.
In our study, the pairs are mined from 275,797 EMR documents of two medical departments (Gynaecology and Respiration). On average, each disease of Gynaecology in our experiments is associated with 24 findings and that of Respiration is 42. For Gynaecology, there are 33 diseases, 305 symptoms, 143 vital signs and 25 test report findings in the PGMs. For Respiration, there are 21 diseases, 263 symptoms, 187 vital signs and 31 test report findings in the PGMs.
3.3.2 Relation Weights Estimation
We experiment with six classical text features as the relation weights in this study.
(1) Occurrence. The weight of finding  $i$  given disease  $j$  is:
$$
w (i; j) = \frac {n (i , j)}{\sum_ {k} n (k , j)}, \tag {1}
$$
where  $n(i,j)$  is the number of co-occurrences of finding  $i$  and disease  $j$ .  $w(i;j)$  is computed by the type of findings.
(2) TF-IDF Feature. Similar to TF-IDF feature in information retrieval, the weight of finding  $i$  given disease  $j$  is:
$$
w (i; j) = n (i, j) * \left(\log \frac {\left| \mathbf {D} \right| + 1}{n _ {i} + 1} + 1\right), \tag {2}
$$
where  $n_i$  is the number of diseases whose EMR documents contain finding  $i$ .
(3) TFC Feature. TFC feature (Salton and Buckley, 1988) is a variant of TF-IDF and it estimates the weight of finding  $i$  given disease  $j$  as:
$$
w (i; j) = \frac {n (i , j) * \log \frac {| \mathbf {D} |}{n _ {i}}}{\sqrt {\sum_ {k} (n (k , j) * \log \frac {| \mathbf {D} |}{n _ {k}}) ^ {2}}}. \tag {3}
$$
(4) TF-IWF Feature. The Term-Frequency Inverse-Word-Frequency (TF-IWF) feature (Basilii et al., 1999) estimates the weight of finding  $i$  given disease  $j$  as:
$$
w (i; j) = n (i, j) * \left(\log \frac {\sum_ {k} t _ {k}}{t _ {i}}\right) ^ {2}, \tag {4}
$$
where  $t_i$  represents the number of occurrences of word  $i$  in the whole training corpus.
(5) CHI Feature. CHI feature  $(\chi^2$  Test) measures how much a term is associated with a class from a statistical view. The CHI feature of finding  $i$  given disease  $j$  is (Yang and Pedersen, 1997):
$$
w (i; j) = \frac {N * (A * D - C * B) ^ {2}}{(A + C) * (B + D) * (A + B) * (C + D)}, \tag {5}
$$
where  $N, A, B, C$  and  $D$  are the number of all documents, the number of documents containing finding  $i$  and belonging to disease  $j$ , the number of documents containing  $i$  but not belonging to  $j$ , the number of documents belonging to  $j$  but not containing  $i$ , and the number of documents not containing  $i$  and not belonging to  $j$ .
(6) Mutual Information. This feature assumes that the higher the strength between a finding and a disease, the higher their mutual information will be. Similar to the definition in CHI feature, this feature is defined as:
$$
w (i; j) \approx \log \frac {A * N}{(A + C) * (A + B)}. \tag {6}
$$
The above features are normalized by disease before applying to the diagnosis inference. By default, the average of the six features is used as the connection weight.
3.3.3 Diagnosis Inference
We propose the Bayesian network ensembles for the diagnosis inference. Specifically, a group of PGMs with the extracted relations and weights are ensembled towards the final predictions.
Firstly, multiple bipartite graphs between disease nodes and each type of finding nodes are derived from the medical knowledge graph. For  $M$  types of findings, there will be  $M$  bipartite graphs. In later experiments,  $M = 3$ , i.e. (disease, symptom), (disease, vital sign) and (disease, test result finding). Based on the findings extracted from EMR document, each bipartite graph can be independently used to infer the disease distribution.
5The undergraduate teaching materials in most of the medical schools in China, authorized by the publisher.
For Bayesian inference, we compute the posterior probability of diseases given the findings in the EMR document extracted by NER:
$$
\Pr (d \mid F ^ {+}, F ^ {-}) = \frac {\Pr (d , F ^ {+} , F ^ {-})}{\Pr (F ^ {+} , F ^ {-})}, d \in \mathbf {D}, \tag {7}
$$
where  $F^{+}$  and  $F^{-}$  are the sets of the positive and the negative findings in the given EMR document, respectively. Following Eq. (7), it is straightforward to get  $\operatorname{Pr}(d|F_{sym}^{+}, F_{sym}^{-})$ ,  $\operatorname{Pr}(d|F_{sign}^{+}, F_{sign}^{-})$  and  $\operatorname{Pr}(d|F_{test}^{+}, F_{test}^{-})$  w.r.t. the predictions based on symptom alone, vital sign alone and test report finding alone. To compute the joint probability  $\operatorname{Pr}(d, F^{+}, F^{-})$  and  $\operatorname{Pr}(F^{+}, F^{-})$ , we refer the readers to the QuickScore method (Heckerman, 1990) and the deduction therein. To speed up computation when a disease is associated with too many positive findings, the variational method on the PGMs is applied (Jordan et al., 1999).
Next, we assemble these bipartite graphs in different ways to get three variants of PGMs (Fig. 1).
(1) Parallel. This method independently performs inference with each type of finding and average their results:
$$
\begin{array}{l} \operatorname * {P r} (d | F ^ {+}, F ^ {-}) = a v g (\operatorname * {P r} (d | F _ {s y m} ^ {+}, F _ {s y m} ^ {-}), \\ \Pr (d | F _ {s i g n} ^ {+}, F _ {s i g n} ^ {-}), \Pr (d | F _ {t e s t} ^ {+}, F _ {t e s t} ^ {-})). \tag {8} \\ \end{array}
$$
Parallel assumes that the ways to diagnose disease are different using different types of entities, and their predictions can complement each other. An extension of Parallel is to perform a weighted sum of the three predictions. For simplicity concerns, we experiment with equal weights in this paper.
(2) Universal. This method mixes all types of findings together into a single network:
$$
\begin{array}{l} \Pr (d \mid F ^ {+}, F ^ {-}) = \tag {9} \\ \operatorname * {P r} (d | F _ {s y m} ^ {+}, F _ {s y m} ^ {-}, F _ {s i g n} ^ {+}, F _ {s i g n} ^ {-}, F _ {t e s t} ^ {+}, F _ {t e s t} ^ {-}). \\ \end{array}
$$
It means that Universal does not distinguish the types of entities and performs the type-free Bayesian inference. Compared with the other two PGM variants, the connections between diseases and findings in Universal are much denser. It assumes that the prediction benefits from the joint inference by seeing more findings of multiple types at the same time.
(3) Cascade. This method constructs the multilayer Bayesian networks with finding types as layers and use the output of the previous layer as the
prior probability for the current layer.
$$
\begin{array}{l} \operatorname * {P r} (d _ {s y m}) = \operatorname * {P r} (d | F _ {s y m} ^ {+}, F _ {s y m} ^ {-}) \\ s. t., d \sim \Pr (d _ {C N N}), \\ \end{array}
$$
$$
\begin{array}{l} \operatorname * {P r} (d _ {s i g n}) = \operatorname * {P r} (d | F _ {s i g n} ^ {+}, F _ {s i g n} ^ {-}) \\ s. t., d \sim \Pr (d _ {s y m}), \\ \end{array}
$$
$$
\begin{array}{l} \operatorname * {P r} (d _ {B N}) = \operatorname * {P r} (d _ {t e s t}) = \operatorname * {P r} (d | F _ {t e s t} ^ {+}, F _ {t e s t} ^ {-}) \\ s. t., d \sim \Pr (d _ {\text {s i g n}}), \tag {10} \\ \end{array}
$$