SlowGuess/ACL_2020_OCR huggingface dataset:
where  $\operatorname{Pr}(d_{CNN})$  is the disease probability distribution computed by the convolutional networks in Sec. 3.2 and  $d \sim \operatorname{Pr}(d_x)$  means that variable  $d$  satisfies prior probability distribution  $\operatorname{Pr}(d_x)$ . Cascade first infers disease with symptoms alone and uses the disease probability from ECNN as priors. Then, it infers disease with vital signs alone and uses the disease probability from symptom-based inference as priors. Finally, it infers disease with test report findings alone and uses the disease probability from the previous output as priors. We present the cascade approach in such order because it shows the best results compared to those in other orders in our experiments. Cascade assumes that each type of entities can be used to refine the previous predictions by incorporating additional information.
The output of the above three PGMs are ensembled, e.g. weighted sum, as the final predictions. In all, the proposed framework takes the raw EMR document and the NER results as input, and outputs the diagnosis predictions.
Although we experiment with three types of entities in this paper, the proposed Bayesian network ensemble method is not limited to these types of entities. It is easy to add more entity types in the proposed method when applicable.
3.4 The Interpretability of BN Ensembles
One of the major contributions of this work is to bring interpretability into automatic diagnosis by stacking the Bayesian network ensembles on top of the convolutional networks. We illustrate how the predictions are explained, i.e. interpretability, by BN with Fig. 2. We use the symptom-based bipartite graph to illustrate for the simplicity concern, and the other types of entities explain the predictions in the same way.
In Fig. 2, if only pharyngalgia is extracted from a patient's EMR, then upper respiratory infection (URI) will be predicted with high probability but the probability of pneumonia and phthisis will
Figure 2: The example of the interpretability of Bayesian network. The connection from disease  $d$  to symptom  $s$  represents that  $d$  has some probability to cause  $s$  to be present. If  $d$  is diagnosed, the detected symptoms from EMR that are connected with  $d$  can be used to explain the diagnosis.
be set to the minimum because both of them are not likely to cause pharyngalgia based on their co-occurrences in the corpus. The proposed method can explain the prediction of URI with symptom pharyngalgia and their co-occurrence times besides the prediction probability.
If pharyngalgia and hemoptysis are both extracted from a patient's EMR, then URI as well as phthisis will be predicted with some positive probability (their rankings depend on both their prior probability and their connection weights to pharyngalgia and hemoptysis), but pneumonia will be predicted with the minimum probability. This is because the noisy-OR gate is used in the Bayesian inference (Heckerman, 1990). The proposed method explains the prediction of URI with the positive finding of symptom pharyngalgia and explains the prediction of phthisis with the positive finding of symptom hemoptysis as well as their co-occurrences.
In this section, we will introduce the data sets we experiment with and the evaluation results.
The proposed framework is evaluated on the real EMR documents (mostly admission records). We have collaborated with several top hospitals in China and we are authorized to conduct experiments with 275,797 EMR documents of two medical departments for the evaluation (see Table 3).<sup>6</sup>
Figure 3: The long-tail distribution of diagnosis. The x-axis indexes the names of diagnosis. The y-axis counts the occurrences of diagnosis in the log scale.
Table 3: The statistics of the data sets. The table represents the document counts by source. # means the number of. " # collected" is the number of the collected EMR documents in the our experiments.
Departments	# collected	# test	# disease
Gynaecology	191,645	606	33
Respiration	84,152	214	21
The collected EMR documents are processed as follows: The main diagnosis in each EMR document is extracted as its disease label. Then, we select the top diseases from the collected EMR documents, which results in 33 diseases from Gynaecology (including Salpingitis, Cervical Carcinoma, Endometritis, Fibroid, etc) and 21 diseases from Respiration (including Upper Respiratory Infection, Chronic Bronchitis, Pneumonia, Asthma, Lung Cancer, etc) that cover over  $90\%$  of all EMR documents. There is a long-tail distribution of EMR documents by diseases as shown in Fig. 3, and each of the selected diseases has over 100 EMR documents for training. The other diseases are discarded in the experiments due to the lack of enough EMR documents to train a trustworthy model. Next, in order to ensure the validity of the disease labels in the test set, we recruit 10 professional physicians to review the labels by evenly sampling EMR documents under each disease. In this way, we collected 606 reviewed EMR documents for Gynaecology and 214 for Respiration as the test set (See disease distribution in supplemental files). The rest EMR documents are used for training. Since we are not given the identity of patient w.r.t. each EMR, the training and the testing sets are considered disjoint. In later experiments, we separately report the performance under both departments. It is more important and difficult to distinguish diseases within the same department than that across departments due to the overlapping symptoms, signs and test report findings among the similar diseases.
Unfortunately, we have not yet obtained the permission from the hospitals to make the evaluation data sets public at this moment because EMR documents are legally protected by the Chinese laws and there is too much sensitive information about the patients and the doctors in them. We are currently working with the hospitals in contributing the benchmark EMR data sets for automatic diagnosis, but it takes time due to the legal issues. We suggest the readers to focus their attention on the contribution of the novel automatic diagnosis framework in this paper.
Table 4: The accuracy of the different diagnosis methods on two medical departments. Top-  $k$  sensitivity is used as the accuracy measurement.
Methods	Gynaecology	Respiration
Top-1	Top-3	Top-1	Top-3
CAML (2018)	58.6%	76.3%	60.7%	82.7%
CNN (2018)	61.0%	82.8%	61.7%	80.8%
ACNN (2018)	62.1%	83.3%	60.7%	84.6%
PGM-C	50.8%	64.6%	26.6%	47.6%
PGM-P	56.1%	69.3%	31.3%	45.3%
PGM-U	56.2%	69.6%	33.6%	57.9%
PGM-E	53.9%	70.2%	28.0%	48.1%
ECNN	68.9%	86.7%	65.8%	81.7%
ECNN-PGM-C	71.4%	88.6%	52.8%	82.7%
ECNN-PGM-U	72.9%	88.6%	59.3%	87.8%
ECNN-PGM-P	73.2%	88.4%	68.2%	87.3%
ECNN-PGM-E	73.4%	88.8%	64.0%	88.3%
We conduct experiments on the collected data sets to evaluate the performance of the framework.
4.2.1 Experimental Settings
In the experiments, we used four CNN towers  $(N = 4)$  w.r.t. CC, HPI, PE and TR, and each tower has three channels with kernel length 3, 4 and 5 (representing 3-grams, 4-grams and 5-grams).
We use Jieba package to perform Chinese word segmentation on the training set and remove the punctuation from the segmentation results. The segmented word corpus is used to train the 100-dimensional word embeddings using the Word2Vec (Mikolov et al., 2013) method (window as 5, min support as 5) implemented in thegensim package. The top 100,000 frequent segmented words consist of the word vocabulary in the embedding layer of ECNN. Thus, the size of the embedding layer is (100000, 100).
Besides, the top 10,000 frequent entities (not segmented words) as well as age and gender are used to construct the one-hot feature into MLP which consists of one hidden dense layer (128 Sigmoid units) due to the efficiency consideration. Similar to Kim (2014), the dropout rate is empirically set to 0.5. By default, we use the average of all six relation weights in the experiments. The final predictions are the average of the three PGM variants. ECNN and PGMs are trained separately offline.
4.2.2 Performance Accuracy
Table 4 shows the Top-  $k$  sensitivity (The micro average of the per-disease Top-  $k$  sensitivity, com
monly used as the accuracy measurement in healthcare studies (Liang et al., 2019).) under two departments. Generally, sensitivity is usually used in binary classification (mostly output yes or no). Similarly, when we are dealing with classification of multi-class rather than binary classification, the proposed automatic diagnosis model outputs the probability distribution over  $K$  diseases (classes) for a given EMR. Suppose there are  $l_{i}$  out of  $n_{i}$  cases, where  $d_{i}$  is included in the Top-  $k$  predictions (ranked by probability) for the  $n_{i}$  EMRs of disease  $d_{i}$ . The Top-  $k$  sensitivity of the proposed model on disease  $d_{i}$  is:  $\frac{l_{i}}{n_{i}}$ . Furthermore, in the overall evaluation of the proposed model on all diseases, we use the micro average of all classes as the overall Top-  $k$  sensitivity:
$$
s e n s i t i v i t y = \frac {\sum_ {i} l _ {i}}{\sum_ {i} n _ {i}}. \tag {11}
$$
CAML (Mullenbach et al., 2018) performs the label-wise attention on top of a CNN model. CNN (Yang et al., 2018) concatenates CC, HPI and TR together before sending to the multi-channel CNN model. ACNN (Girardi et al., 2018) incorporates the gram-level attention with a CNN model. The empirical settings of hyper parameters are selected from the original papers. Besides, they share the same training set, training epochs, learning rate and batch size with the proposed methods.
Among the proposed methods,  $PGM^{-*}(-C, -P, -U$  and  $-E$  represent Cascade, Parallel, Universal and Ensemble, respectively) are the methods that solely relies on the Bayesian networks which use the disease distribution in the training set as the prior probability. ECNN is the proposed method without the BN ensembles.  $ECNN-PGM^{-*}$  are the combined methods while  $ECNN-PGM-E$  is the proposed method with ECNN and Bayesian network ensembles in Figure 1. According to the results: (1) Most of the proposed methods  $ECNN-PGM^{-*}$  outperform the previous automatic diagnosis methods, which shows the effectiveness of the proposed methods. (2) ECNN outperforms CNN due to the incorporation of medical entities. Jointly modeling with free texts and medical entities brings extra accuracy performance compared with modeling with only either one. (3) Stacking Bayesian Networks on top of the neural networks is very likely to further improve the performance, especially with the ensemble of the predictions from multiple PGMs.
<sup>8</sup>https://radimrehurek.com/gensim/
(a) Gynaecology
(b) Respiration
Figure 4: Top-1 sensitivity by diseases.
4.2.3 Error Analysis
Fig. 4 shows the Top-1 sensitivity on some diseases. The performances across diseases are quite different. For example, the Top-1 sensitivity of Salpingitis is  $100\%$  but that of Endometriosis is  $29\%$  in the evaluation. Salpingitis can be identified by combining general symptoms and ultrasonic exam results. However, from the perspective of physicians, Endometriosis is difficult to diagnose by nature because it shares common symptoms like dysmenorrhea and irregular menstruation with other Gynecologic diseases. These shared findings misguide the classifier towards other similar diseases. Similarly, among the respiratory diseases, patients with Pulmonary Embolism, Respiratory Failure and Bronchiectasia share symptom dyspnea which makes it difficult to distinguish between them. In contrast, Upper Respiratory Infection (URI) is easy to diagnose because it causes throat pain and rhinorrhea unlike the other respiratory diseases.
Based on the analysis, the diagnosis performance of a disease is higher if it shares less findings with other diseases or it has more specific findings.
4.2.4 Interpretability
The interpretability is reflected on the observed findings in the EMR that connect to the predicted disease in the medical knowledge graph as well as their co-occurrences. We generate the prediction explanation with the following template: The patient is diagnosed as disease  $d$  because  $(s)he$  is suffering from symptom  $s_i$ , and  $(s)he$  has the vital sign of  $v_j$ , and the lab test (or PACS report) shows  $(s)he$  has  $t_k$ . Besides,  $s_i$ ,  $v_j$  and  $t_k$  have been found on the patients of  $d$  for  $n_i$ ,  $n_j$ ,  $n_k$  times, respectively, in the previous EMR documents that support this diagnosis.
Since the extracted relations in the medical knowledge graph are reviewed by the certificated physicians, the validity of explanation is guaranteed from the clinical perspective. We randomly select 50 testing samples per department whose Top-1 diagnosis prediction is correct and generate the explanation for the diagnosis prediction with
Figure 5: The accuracy of ECNN-PGM-E using different types of features. Gyn and Res represent gynaecology and respiration, respectively. MI and Occ are mutual information and occurrence, respectively.
the above template. The explanation is evaluated by three certificated physicians. The evaluation is subjective, but all of them agree that the prediction is well-supported by the generated explanation.
4.2.5 Feature Importance
Figure 5 shows the accuracy performance using different types of features. We can see that in this evaluation, TFC, TF-IDF and the average of all features are likely to lead to higher accuracy compared to the other features where the accuracy of Top-3 prediction is over  $88\%$ .
In all, the above experiments prove that the proposed framework can improve the accuracy of automatic diagnosis and bring reasonable interpretability into the predictions in the same time.
In this paper, we investigate the problem of automatic diagnosis with EMR documents for clinical decision support. We propose a novel framework that stacks the Bayesian Network ensembles on top of the Entity-aware Convolutional Neural Networks. The proposed design brings interpretability into the predictions, which is very important for the AI-empowered healthcare, without compromising the accuracy of convolutional networks. The evaluation conducted on the real EMR documents from hospitals validates the effectiveness of the proposed framework compared to the baselines in automatic diagnosis with EMR.
We thank all the professional physicians led by Dr. Shi and Dr. Hu who have contributed in the annotation tasks in our experiments.
Padmanabhan Anandan, Yan Huang, Kazumi Nishikawa, BBorie Park, Eric S. Sullivan, Jingyu Wang, and Xu Shan. 2019. AI in health care: Capacity, capability, and a future of active health in Asia. MIT Technology Review Insights, pages 1-25.
Roberto Basili, Alessandro Moschitti, and Maria Teresa Pazienza. 1999. A text classifier based on linguistic processing. In *IJCAI Workshop on Machine Learning and Information Filtering*, Stockholm, Sweden.
Tal Baumel, Jumana Nassour-Kassis, Raphael Cohen, Michael Elhadad, and Noémie Elhadad. 2017. Multi-label classification of patient notes a case study on icd code assignment. In AAAI Workshops, pages 409-416.
Eta S. Berner. 2007. Clinical Decision Support Systems. Springer.
Jun Chen, Jingbo Zhou, Zhenhui Shi, Bin Fan, and Chengliang Luo. 2019. Knowledge abstraction matching for medical question answering. In IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 342-347.
Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F Stewart, and Jimeng Sun. 2016a. Doctor AI: Predicting clinical events via recurrent neural networks. In Machine Learning for Healthcare Conference, pages 301-318.
Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. 2016b. RETAIN: An interpretable predictive model for healthcare using reverse time attention mechanism. In NeurIPS, pages 3504-3512.
Dai Dai, Xinyan Xiao, Yajuan Lyu, Shan Dou, Qiaoqiao She, and Haifeng Wang. 2019. Joint extraction of entities and overlapping relations using position-attentive sequence labeling. In AAAI, Honolulu, Hawaii, USA.
Andre Esteva, Alexandre Robicquet, Bharath Ramsundar, Volodymyr Kuleshov, Mark DePristo, Katherine Chou, Claire Cui, Greg Corrado, Sebastian Thrun, and Jeff Dean. 2019. A guide to deep learning in healthcare. Nature Medicine, 25:24-29.
Ivan Girardi, Pengfei Ji, An phi Nguyen, Nora Hollenstein, Adam Ivankay, Lorenz Kuhn, Chiara Marchiori, and Ce Zhang. 2018. Patient risk assessment and warning symptom detection using deep attention-based neural networks. In EMNLP Workshop, pages 139-148, Brussels, Belgium.
David Heckerman. 1990. A tractable inference algorithm for diagnosing multiple diseases. Machine Intelligence and Pattern Recognition, 10:163-171.
Wei Jia, Dai Dai, Xinyan Xiao, and Hua Wu. 2019. ARNOR: Attention regularization based noise reduction for distant supervision relation classification. In ACL, pages 1399-1408, Florence, Italy.
Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. 1999. An introduction to variational methods for graphical models. Machine Learning, 37:183-233.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP, pages 1746-1751, Doha, Qatar.
Theresa A Koleck, Caitlin Dreisbach, Philip E Bourne, and Suzanne Bakken. 2019. Natural language processing of symptoms documented in free-text narratives of electronic health records: a systematic review. Journal of the American Medical Informatics Association, pages 364-379.
Christy Li, Dimitris Konomis, Graham Neubig, Pengtao Xie, Carol Cheng, and Eric Xing. 2017. Convolutional neural networks for medical diagnosis from admission notes. In arXiv.
Huiying Liang, Brian Y. Tsui, Hao Ni, Carolina C. S. Valentim, Sally L. Baxter, and et al. 2019. Evaluation and accurate diagnoses of pediatric diseases using artificial intelligence. Nature Medicine, 25:433-438.
Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen A.W.M. van der Laak, Bram van Ginneken, and Clara I. Sanchez. 2017. A survey on deep learning in medical image analysis. Medical Image Analysis, 42:60-88.
Qianlong Liu, Zhongyu Wei, Baolin Peng, Xiangying Dai, Huaixiao Tou, Ting Chen, Xuanjing Huang, and Kam fai Wong. 2018. Task-oriented dialogue system for automatic diagnosis. In ACL, pages 201-207, Melbourne, Australia.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representation of words and phrases and their compositionality. In NeurIPS, pages 3111--3119.
James Mullenbach, Sarah Wiegrefe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. 2018. Explainable prediction of medical codes from clinical text. In NAACL, pages 1101-1111, New Orleans, Louisiana, USA.
Aaditya Prakash, Siyuan Zhao, Sadid A. Hasan, Vivek Datla, Kathy Lee, Ashequl Qadir, Joey Liu, and Oladimeji Farri. 2017. Condensed memory networks for clinical diagnostic inferencing. In AAAI, pages 3274-3280.
Gerard Salton and Christopher Buckley. 1988. Termweighting approaches in automatic text retrieval. Information Processing & Management, 24(5):513-523.
Ying Sha and May D. Wang. 2017. Interpretable predictions of clinical outcomes with an attention-based recurrent neural network. In ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, pages 233-240, Boston, MA, USA.
Xue Shi, Yingping Yi, Ying Xiong, Buzhou Tang, Qingcai Chen, Xiaolong Wang, Zongcheng Ji, Yaoyun Zhang, and Hua Xu. 2019. Extracting entities with attributes in clinical text via joint deep learning. Journal of the American Medical Informatics Association, pages 1584-1591.
Yiming Yang and Jan O. Pedersen. 1997. A comparative study on feature selection in text categorization. In ICML, pages 412--420, Nashville, TN, USA.
Zhongliang Yang, Yongfeng Huang, Yiran Jiang, Yuxi Sun, Yu-Jin Zhang, and Pengcheng Luo. 2018. Clinical assistant diagnosis for electronic medical record based on convolutional neural network. Scientific Reports, 8(6329).
Shiyue Zhang, Pengtao Xie, Dong Wang, and Eric P. Xing. 2017. Medical diagnosis from laboratory tests by combining generative and discriminative learning. In *arxiv*.