SlowGuess/NAACL_2021_OCR huggingface dataset:
The diagnostic results are shown in Table 2. From the results, we can see our model performs better than all the compared models and can achieve about  $99\%$  accuracy in the diagnosis of lymphedema, which exceeds the comparison models by  $3\% -5\%$  in precision, recall, and F1. Based on our model, the categories of clauses and entities can be distinguished correctly, which demonstrates that the clustering information contained in the pseudolabels is correctly learned by our multi-granularity model. This result indicates that the accuracy of our method in the diagnosis of lymphedema is in line with clinical requirements. Since our goal is to make the model really help doctors in clinical practice with reliable medical explanations, we will focus on the performance of the counterfactual extraction of the supporting facts for the diagnosis that follows. Please refer to Appendix A.4 for the effectiveness of our model in diagnosis on the benchmark data.
4.2 Counterfactual Extraction Result
Automatic Result Table 3 shows the automatic evaluation results of the supporting facts extraction. Since the identified keywords are difficult to accurately cover the features for diagnosis and models have different adaptability to various diseases, the performance is distinguishing on different diseases. Compared with other models, the counterfactual
Model	Secondary Lymphedema	Primary Lymphedema	Chylous Reflux Lymphedema
hit@1	hit@3	hit@5	hit@1	hit@3	hit@5	hit@1	hit@3	hit@5
Self-Attention	5.45%	36.36%	50.91%	13.64%	45.45%	54.55%	4.76%	9.52%	33.33%
PostKS	9.09%	43.64%	60.00%	9.09%	40.91%	54.55%	0.00%	14.29%	19.05%
Node-Mask	25.45%	52.73%	69.09%	22.73%	31.82%	54.55%	9.52%	19.05%	23.81%
Edge-Mask	36.36%	61.82%	70.91%	22.73%	40.91%	50.00%	61.90%	66.67%	76.19%
Table 3: The automatic evaluation for the extraction of diagnostic supporting facts for three types of lymphedema.
Model	Secondary Lymphedema	Primary Lymphedema	Chylous Reflux Lymphedema	Others
C	T	C	T	C	T	C	T
Self-Attention	0.85	0.83	0.75	0.95	0.62	0.33	1.09	0.76
PostKS	0.64	0.96	0.66	0.82	0.67	0.48	0.57	0.42
Node-Mask	1.44	0.91	1.11	1.02	1	0.71	1.67	1.24
Edge-Mask	1.67	1.22	1.36	1.11	1.33	1.38	1.67	1.52
Table 4: The human evaluation for the extraction of diagnostic supporting facts for each type. In the table, C stands for comprehensiveness and T stands for trustworthiness.
based methods, especially the Edge-Mask method, has an advantage in accuracy and robustness on the whole. Hit@1 shows that the Edge-Mask can locate key facts more quickly than the comparison methods and hit@5 shows that it achieves over  $70\%$  accuracy on secondary lymphedema and chylous reflux lymphedema. In the comparison of different lymphedema, other methods have a greater performance degradation, and only the Edge-Mask maintains high accuracy in various diseases, indicating that the Edge-Mask method is highly robust to different diseases.
Human Result Table 4 shows the results of the human evaluation of the four categories of diagnosis. Compared with other methods, the counterfactual-based methods have great advantages in comprehensiveness, which indicates that our method can focus more on useful medical information and eliminate invalid noise in the EMR. The fourth category requires focus. This category includes all non-lymphedema medical records, and its diseases are diverse and complex. It can be seen that the method of counterfactual reasoning has strong performance in this type in terms of comprehensiveness and credibility, indicating that our method is truly independent of the type of disease and suitable for large-scale promotion.
4.3 Effectiveness of Clustering Labels
Table 2 shows the ablation experiment results for the clustering labels. For the experiment without corresponding labels, we used a classifier with random initialization parameters for classification,
which can reflect the expectation of the ability to encode medical features of the model. The results show that both the clause label and entity label can improve the accuracy of diagnosis by about  $1\%$  on the basis of over  $96\%$  accuracy. Since we use the same encoder to encode the three granular texts of the sentence, clause and entity, the addition of clause labels also improves the accuracy of entity classification and vice versa. The result indicates that the introduction of cluster tags enhances the expression of medical information in the model and enables the model to better extract and utilize relevant medical knowledge from irregular text.
4.4 Advanced Analysis
Results in Primary Lymphedema Since the diagnosis of primary lymphedema is mainly diagnosed by excluding other types of lymphedema, the keywords we established are not standardized in the EMR, the performance of all models in Table 3 has a significant decline and only be used for comparison. And the performance in human evaluation is consistent with other diseases in Table 4.
Results in Chylous Reflux Lymphedema Except for Edge-Mask, the performance of the other methods on chylous reflux lymphedema has dropped significantly. Since this type of EMR only accounts for  $4\%$  of the dataset, the models based on frequency statistics are difficult to capture key features. And Edge-Mask, using counterfactual intervention to obtain causal relation, is disease-independent and can adapt to few data.
Figure 4: An example of a trustworthy supporting facts graph extracted by Edge-Mask. The keywords "cancer", "surgery" and "chemotherapy" in this result meet the diagnostic criteria for secondary lymphedema in Appendix B.3.
Node-Mask and Edge-Mask Edge-Mask is included in Node-Mask. Masking the feature of a node will inevitably reduce the flow of information on all connected edges. So compared to NodeMask, Edge-Mask is a fine-grained counterfactual intervention. For Node-Mask, the flow of multigranularity information between nodes will be truncated. For example, when a clause node is masked, the entity features belonging to it are truncated together. Therefore, Node-Mask has a weaker performance than Edge-Mask.
4.5 Visual Presentation of Results
Figure 4 is an example randomly obtained from the test set. In this graph, each node represents a clause that contains the entities used to describe the symptoms of the disease and the edges represent the connection between them. All the aforementioned features constitute a hierarchical supporting graph to provide effective help for doctors' diagnosis. As we can see, our model successfully extracted the patient's history of cancer, surgery and chemotherapy, which can clearly indicate that the patient is suffering from secondary lymphedema. This shows that the supporting facts we extracted are effective. We provide a comparison of the extraction results of different models in Appendix A.3.
Figure 5 shows an example of the visualization of the Edge-Mask matrix. It can be seen that most of the edges have been masked, and only the edges from two key feature nodes have been preserved. This proves that our method can effectively filter noisy features and extract supporting facts.
Figure 5: An example of visualization of Edge-Mask matrix. The same as the adjacency matrix, the rows and columns in the figure correspond to the nodes in the graph, and each grid in the figure represents a value in the Edge-Mask matrix.
Explainable Diagnosis with EMR It is necessary to provide explainability for automatic diagnosis systems. CAML (Mullenbach et al., 2018) provides explanations with the spans having the high attention weights in the text sequence and (Feng et al., 2020) calculates a threshold for attention selection. AdaCare (Ma et al., 2020a) calculate the average importance weights in the overall dataset to obtain symptoms strongly associated with the diseases. These works focus on correlations based on attention and ignore causality between features and diagnosis.
Document Modeling with Graph Network Document modeling with graph network has been widely used in text classification (Yao et al., 2019), multi-hop reading comprehension (Cao et al., 2019) and abstract extraction (Wang et al., 2020). An EMR can also be considered as a document. There are two main ways to structure a document into a graph, based on the entity (Qiu et al., 2019) or based on the structure of the document (Zheng et al., 2020). (Tu et al., 2019) considers the integration of documents and entities as heterogeneous nodes in the graph network, and (Fang et al., 2019) propose a hierarchical model that combines document structure and entity structure. We used a multi-granularity hierarchical graph network to model the EMR documents.
Counterfactual Reasoning Providing explanations based on counterfactual reasoning has a long
history (Lewis, 1973; Woodward, 2005). In recent years, (Oberst and Sontag, 2019) introduces a kind of structural causal model to genera counterfactual trajectories in a synthetic environment of sepsis management. (Lin et al., 2020) presents a patient simulator to generate informative counterfactual response in the disease diagnosis. (Lenis et al., 2020) identifies salient regions of a medical image by measuring the effect of local counterfactual image-perturbations. We use counterfactual reasoning in EMRs to provide explanations for diagnosis.
In this paper, we propose a counterfactual multi-granularity graph supporting facts extraction (CMGE) method for the irregular EMR without an external medical knowledge base. Based on this model, we can correctly diagnose lymphedema. The proposed counterfactual-based approach can discover the causal relationship between symptoms and diagnosis. The results of supporting fact extraction show that our method has strong robustness and can maintain accuracy in various diseases and even in categories with few data resources. In the future, we will introduce multi-modal into the model such as radiology images to discover more medical knowledge from EMRs.
This work is supported by the National Key Research and Development Program of China (No.2018YFB1005104) and the Key Research Program of the Chinese Academy of Sciences (ZDBSSW-JSC006).
Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In NIPS, pages 2787-2795.
Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao, Shengping Liu, and Weifeng Chong. 2020. Hypercore: Hyperbolic and co-graph representation for automatic ICD coding. In ACL, pages 3105-3114. Association for Computational Linguistics.
Yu Cao, Meng Fang, and Dacheng Tao. 2019. BAG: bidirectional attention entity graph convolutional network for multi-hop reasoning question answering. In *NAACL-HLT* (1), pages 357–362. Association for Computational Linguistics.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gülcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, pages 1724-1734. ACL.
Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F. Stewart, and Jimeng Sun. 2016. Doctor AI: predicting clinical events via recurrent neural networks. In MLHC, volume 56 of JMLR Workshop and Conference Proceedings, pages 301-318. JMLR.org.
Edward Choi, Cao Xiao, Walter F. Stewart, and Jimeng Sun. 2018. Mime: Multilevel medical embedding of electronic health records for predictive healthcare. In NeurIPS, pages 4552-4562.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT* (1), pages 4171–4186. Association for Computational Linguistics.
Hang Dong, Víctor Suárez-Paniagua, William Whiteley, and Honghan Wu. 2020. Explainable automated coding of clinical notes using hierarchical label-wise attention networks and label embedding initialisation. CoRR, abs/2010.15728.
Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu. 2019. Hierarchical graph network for multi-hop question answering. CoRR, abs/1911.03631.
Jinyue Feng, Chantal Shaib, and Frank Rudzicz. 2020. Explainable clinical decision support from text. In EMNLP (1), pages 1478-1489. Association for Computational Linguistics.
Ivan Girardi, Pengfei Ji, An-phi Nguyen, Nora Hollenstein, Adam Ivankay, Lorenz Kuhn, Chiara Marchiori, and Ce Zhang. 2018. Patient risk assessment and warning symptom detection using deep attention-based neural networks. In Louhi@EMNLP, pages 139-148. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In *ICLR (Poster)*.
Dimitrios Lenis, David Major, Maria Wimmer, Astrid Berg, Gert Sluiter, and Katja Bühler. 2020. Domain aware medical image classifier interpretation by counterfactual impact analysis. In MICCAI (1), volume 12261 of Lecture Notes in Computer Science, pages 315-325. Springer.
S. M. Levine. 2017. Lymphedema: Complete medical and surgical management. *Plastic & Reconstructive Surgery*, 139(3):772.
David Lewis. 1973. Counterfactuals, blackwells.
Fei Li and Hong Yu. 2020. ICD coding from clinical text using multi-filter residual convolutional neural network. In AAAI, pages 8180-8187. AAAI Press.
Rongzhong Lian, Min Xie, Fan Wang, Jinhua Peng, and Hua Wu. 2019. Learning to select knowledge for response generation in dialog systems. In *IJCAI*, pages 5081-5087. ijcai.org.
Junfan Lin, Ziliang Chen, Xiaodan Liang, Keze Wang, and Liang Lin. 2020. Learning reinforced agents with counterfactual simulation for medical automatic diagnosis. CoRR, abs/2003.06534.
Luchen Liu, Haoran Li, Zhiting Hu, Haoran Shi, Zichang Wang, Jian Tang, and Ming Zhang. 2019. Learning hierarchical representations of electronic health records for clinical outcome prediction. In AMIA. AMIA.
Fenglong Ma, Quanzeng You, Houping Xiao, Radha Chitta, Jing Zhou, and Jing Gao. 2018. KAME: knowledge-based attention model for diagnosis prediction in healthcare. In CIKM, pages 743-752. ACM.
Liantao Ma, Junyi Gao, Yasha Wang, Chaohe Zhang, Jiangtao Wang, Wenjie Ruan, Wen Tang, Xin Gao, and Xinyu Ma. 2020a. Adacare: Explainable clinical health status representation learning via scale-adaptive feature extraction and recalibration. In AAAI, pages 825-832. AAAI Press.
Liantao Ma, Chaohe Zhang, Yasha Wang, Wenjie Ruan, Jiangtao Wang, Wen Tang, Xinyu Ma, Xin Gao, and Junyi Gao. 2020b. Concare: Personalized clinical feature embedding via capturing the healthcare context. In AAAI, pages 833-840. AAAI Press.
James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. 2018. Explainable prediction of medical codes from clinical text. In NAACL-HLT, pages 1101-1111. Association for Computational Linguistics.
Michael Oberst and David A. Sontag. 2019. Counterfactual off-policy evaluation with gumbel-max structural causal models. In ICML, volume 97 of Proceedings of Machine Learning Research, pages 4881-4890. PMLR.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, pages 8024-8035.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations.
Lin Qiu, Yunxuan Xiao, Yanru Qu, Hao Zhou, Lei Li, Weinan Zhang, and Yong Yu. 2019. Dynamically fused graph network for multi-hop reasoning. In ACL (1), pages 6140-6150. Association for Computational Linguistics.
Mike Schuster and Kuldip K. Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Trans. Signal Process., 45(11):2673-2681.
Ying Sha and May D. Wang. 2017. Interpretable predictions of clinical outcomes with an attention-based recurrent neural network. In *BCB*, pages 233–240. ACM.
Ming Tu, Guangtao Wang, Jing Huang, Yun Tang, Xiaodong He, and Bowen Zhou. 2019. Multi-hop reading comprehension across multiple documents by reasoning over heterogeneous graphs. In ACL (1), pages 2704-2713. Association for Computational Linguistics.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In ICLR (Poster). OpenReview.net.
Sahil Verma, John P. Dickerson, and Keegan Hines. 2020. Counterfactual explanations for machine learning: A review. CoRR, abs/2010.10596.
Danqing Wang, Pengfei Liu, Yining Zheng, Xipeng Qiu, and Xuanjing Huang. 2020. Heterogeneous graph neural networks for extractive document summarization. In ACL, pages 6209-6219. Association for Computational Linguistics.
James Woodward. 2005. *Making things happen: A theory of causal explanation*. Oxford university press.
Keyang Xu, Mike Lam, Jingzhi Pang, Xin Gao, Charlotte Band, Piyush Mathur, Frank Papay, Ashish K. Khanna, Jacek B. Cywinski, Kamal Maheshwari, Pengtao Xie, and Eric P. Xing. 2019. Multimodal machine learning for automated icd coding. volume 106 of Proceedings of Machine Learning Research, pages 197-215, Ann Arbor, Michigan. PMLR.
Yuan Xu, Seungwon Lee, Elliot Martin, Adam G. D'souza, Chelsea T.A. Doktorchik, Jason Jiang, Sangmin Lee, Cathy A. Eastwood, Nowell Fine, Brenda Hemmelgarn, Kathryn Todd, and Hude Quan. 2020. Enhancing icd-code-based case definition for heart failure using electronic medical record data. Journal of Cardiac Failure, 26(7):610-617.
Xuefeng Lu Xuehong Wan. 2013. Diagnostics. Beijing: People's Health Publishing House.
Zhongliang Yang, Yongfeng Huang, Yiran Jiang, Yuxi Sun, Yu-Jin Zhang, and Pengcheng Luo. 2018. Clinical assistant diagnosis for electronic medical record based on convolutional neural network. Scientific reports, 8(1):1-9.
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Graph convolutional networks for text classification. In AAAI, pages 7370-7377. AAAI Press.
Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. 2019. Gnnexplainer: Generating explanations for graph neural networks. In NeurIPS, pages 9240-9251.
Quan Yuan, Jun Chen, Chao Lu, and Haifeng Huang. 2020. The graph-based mutual attentive network for automatic diagnosis. In *IJCAI*, pages 3393–3399. ij-cai.org.
Yuhao Zhang, Yuhui Zhang, Peng Qi, Christopher D. Manning, and Curtis P. Langlotz. 2020. Biomedical and clinical english model packages in the stanza python nlp library. arXiv preprint arXiv:2007.14640.
Bo Zheng, Haoyang Wen, Yaobo Liang, Nan Duan, Wanxiang Che, Daxin Jiang, Ming Zhou, and Ting Liu. 2020. Document modeling with graph attention networks for multi-grained machine reading comprehension. In ACL, pages 6708-6718. Association for Computational Linguistics.
To implement our model, we use the tokenizer of BERT (Devlin et al., 2019) to obtain the tokens of the EMR text sequence. For BiGRU (Cho et al., 2014) encoder, the embedding dimension is 300 and the hidden dimension is 256 with two layers. In graph reasoning, we use 2 multi-heads GAT layers with 8 heads. The input dimension of GAT is 1024 and the output dimension is 128. For counterfactual reasoning, we fix the parameters of all the diagnostic models and only optimized the matrix of Edge-Mask or Node-Mask. The hyper-parameters can be set to any possible value based on the tuning. With the manual tuning for diagnostic accuracy, except for  $\lambda_5$ , all the hyper-parameters of the loss function are set to 1 in the experiment, and  $\lambda_5$  is set to 0.1 for Node-Mask and 0.005 for Edge-Mask. We trained on the diagnostic model for 20 epochs and do counterfactual training on each sample for 200 epochs. Our model has a total of 16.7M parameters and can easily train and infer in Titan XP. Since we are not doing parallel processing, counterfactual reasoning is the most consuming, and it takes 7 seconds for each instance.
A.2 PostKS
We modified the PostKS (Lian et al., 2019) model to this task. In order to enhance the accuracy of supporting facts extraction, except the diagnosis
Figure 6: An overview of modified PostKS.
label, we use some additional diagnostic descriptions related to the disease, which are shown in "Diagnosis" in Figure 1.
Figure 6 shows the overview of variational inference model. All the clauses and the diagnosis are encoded by BiGRU and we take the last hidden state  $\pmb{h}_n$  as the feature sequence  $C = [c_1, c_2, \dots, c_n]$  for the clauses and the feature  $d$  for diagnosis. Based on these, we can calculate the posterior distribution as:
$$
p (\boldsymbol {c} = \boldsymbol {c} _ {i} | \boldsymbol {C}, \boldsymbol {d}) = \frac {\exp (\boldsymbol {c} _ {i} \cdot \boldsymbol {d})}{\sum_ {j = 1} ^ {N} \exp (\boldsymbol {c} _ {j} \cdot \boldsymbol {d})} \tag {11}
$$
where  $N$  is the number of clauses,  $c_{i}$  is the feature of the i-th clause, and for prior distribution, we calculate as:
$$
p \left(\boldsymbol {c} = \boldsymbol {c} _ {i} \mid \boldsymbol {c} _ {i}, \boldsymbol {c} _ {k}\right) = \frac {\exp \left(\boldsymbol {c} _ {i} \cdot \boldsymbol {c} _ {k}\right)}{\sum_ {j = 1} ^ {N} \exp \left(\boldsymbol {c} _ {j} \cdot \boldsymbol {c} _ {k}\right)} \tag {12}
$$
Then use the average pooling to obtain the self-attention weight  $p(\boldsymbol{c} = \boldsymbol{c}_i | \boldsymbol{C})$  of each clause and optimize:
$$
\begin{array}{l} \mathcal {L} _ {c} = \lambda_ {d} \mathcal {L} _ {\text {d i a g n o s i s}} \\ + \lambda_ {k} \sum_ {N} ^ {i = 1} p (\boldsymbol {c} = \boldsymbol {c} _ {i} | \boldsymbol {C}, \boldsymbol {d}) \log \frac {p (\boldsymbol {c} = \boldsymbol {c} _ {i} | \boldsymbol {C} , \boldsymbol {d})}{p (\boldsymbol {c} = \boldsymbol {c} _ {i} | \boldsymbol {C})} \tag {13} \\ \end{array}
$$
where  $\lambda_{d},\lambda_{k}$  are hyper-parameters and  $\mathcal{L}_{diagnosis}$  is the cross-entropy loss for diagnosis result.
A.3 Result Comparison
Table 7 shows two examples of supporting facts extraction results. For Secondary Lymphoma, it can be seen that except for PostKS, all other methods can find critical features. PostKS discovered the word "lymphedema" since it is highly correlated with the diagnosis text. The result indicates that
Model	AUC	F1	P@5
Macro	Micro	Macro	Micro
BiGRU (Mullenbach et al., 2018)	82.8	86.8	48.4	54.9	59.1
CNN (Mullenbach et al., 2018)	87.6	90.7	57.6	62.5	62.0
CAML (Mullenbach et al., 2018)	87.5	90.9	53.2	61.4	60.9
DR-CAML (Mullenbach et al., 2018)	88.4	91.6	57.6	63.3	61.8
MultiResCNN (Li and Yu, 2020)	89.9	92.8	60.6	67.0	64.1
HyperCore (Cao et al., 2020)	89.5	92.9	60.9	66.3	63.2
MultiResCNN*	89.3	92.2	59.3	66.2	62.8
BiGRU + MHG	88.9	92.5	57.8	66.6	64.1
MultiResCNN + MHG	90.2	93.1	60.9	67.5	64.7
Table 5: The experimental result on benchmark data. MultiResCNN* represents the result of MultiResCNN on our pre-processed hierarchical structured data. BiGRU, CNN, CAML and DR-CAML are the most frequently compared baselines for this task. MultiResCNN and HyperCore are currently strong and effective baselines.
the posterior information has worked, but it cannot provide an explanation for the diagnosis. The Edge-Mask discovered the "swelling after surgery immediately", which is the best support for the diagnosis of secondary lymphedema, indicating the effectiveness of it. For Chylous Reflux Lymphedema, only Self-Attention and Edge-Mask find critical information like "milky white liquid". Compared with Self-Attention, Edge-Mask has a more complete description of the supporting facts.
A.4 Evaluation on benchmark data
We didn't find any benchmarks on the task of directly extracting supporting facts from EMRs without other knowledge. To better prove the performance of our model, we have done experiments on the English EMR benchmark "MIMIC-III-50" (Mullenbach et al., 2018) for the task of assigning ICD codes to EMRs. This task assigns multiple codes to EMRs from 50 labels. Compared to our diagnosis of four types of EMRs, the difficulty is obvious.