SlowGuess/NAACL_2021_OCR huggingface dataset:
Haoran Wu $^{1,2}$ , Wei Chen $^{1}$ , Shuang Xu $^{1}$  and Bo Xu $^{1,2}$
$^{1}$ Institute of Automation, Chinese Academy of Sciences,
Beijing, 100190, China
$^{2}$ School of Artificial Intelligence, University of Chinese Academy of Sciences,
Beijing, 100049, China
{wuhaoran2018, w.chen, shuang.xu, xubo} @ia.ac.cn
Providing a reliable explanation for clinical diagnosis based on the Electronic Medical Record (EMR) is fundamental to the application of Artificial Intelligence in the medical field. Current methods mostly treat the EMR as a text sequence and provide explanations based on a precise medical knowledge base, which is disease-specific and difficult to obtain for experts in reality. Therefore, we propose a counterfactual multi-granularity graph supporting facts extraction (CMGE) method to extract supporting facts from the irregular EMR itself without external knowledge bases in this paper. Specifically, we first structure the sequence of the EMR into a hierarchical graph network and then obtain the causal relationship between multi-granularity features and diagnosis results through counterfactual intervention on the graph. Features having the strongest causal connection with the results provide interpretive support for the diagnosis. Experimental results on real Chinese EMRs of the lymphedema demonstrate that our method can diagnose four types of EMRs correctly, and can provide accurate supporting facts for the results. More importantly, the results on different diseases demonstrate the robustness of our approach, which represents the potential application in the medical field<sup>1</sup>.
Electronic Medical Record (EMR) based diagnosis has attracted extensive attention due to its comprehensive historical information and clinical descriptions with the development of natural language processing and medical informatics (Yang et al., 2018; Choi et al., 2018; Liu et al., 2019; Dong et al., 2020; Ma et al., 2020b). The application of deep learning in medicine requires adequate medical explanations for the result. Specific to the diagnosis of EMR, the model needs to provide the text description supporting the diagnosis results.
① Age: 49
$②$  Gender:woman
$③$  Document: In December 2011, the patient underwent surgical treatment for endometrial cervical cancer in a local hospital, and received radiotherapy 12 times after surgery. Regular reexamination, there are no signs of tumor recurrence...(患者于2011年12月因子宫内膜宫颈癌于当地医院行手术治疗，术后行放射治疗12次。定期复查，未见肿瘤复发征象.....)
④ Diagnosis: Secondary lymphedema of right lower limb, after endometrial cancer surgery(右下肢继发性淋巴水肿,子宫内膜癌术后)
Supporting Facts: received radiotherapy 12 times after surgery(术后行放射治疗12次)
Figure 1: An example of EMR. We consolidated the various parts of the EMR into a single document as input and our goal is to extract supporting facts at the granularity of the clause.
As shown in Figure 1, an irregular EMR is a document of disease-related information, including symptoms, history of the disease, preliminary examination results, and so on, which is disordered and sparse with meaningless noisy text. Existing methods provide explanation through medical entities (Yuan et al., 2020), text spans (Mullenbach et al., 2018) and the weights of external knowledge (Ma et al., 2018). The entity is critical to the diagnosis (Sha and Wang, 2017; Girardi et al., 2018), but for the medical explanation, it cannot provide specific information of symptoms (such as positive or negative). And the form of the span is too fragmented and lacks readability. Therefore, the clause as a more informative and readable representation is needed to be combined above the level of entities.
Most of the previous methods provide reliable explanations for diagnosis by calculating the similarity with an external medical knowledge base (ICD $^2$  and CCS $^3$ ) (Xu et al., 2019, 2020). KAME
<sup>1</sup>The code is available at https://github.com/CKRE/CMGE
$^{2}$ https://www.cdc.gov/nchs/icd/icd10cm.htm
<sup>3</sup>https://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1942-1955 June 6-11, 2021. ©2021 Association for Computational Linguistics
(Ma et al., 2018) uses the weights of the nodes in the introduced knowledge graph to provide explanations. Depending on the hierarchical relations in the database, GMAN (Yuan et al., 2020) builds a disease hierarchy graph and a causal graph to find critical entities. However, a trusted medical knowledge base requires a mass of expertise in different fields to build, and it may be incomplete or erroneous in practical clinical applications. So far, how to extract supporting facts from the EMR itself without an external medical knowledge base is still a problem.
Counterfactual reasoning provides a link between what could have happened when inputs had been changed (Verma et al., 2020). Doctors usually make a judgment based on several related symptoms during diagnosing a disease. In this regard we can consider a question: will a doctor make a misdiagnosis without one of the critical symptoms? The result is clear. In a counterfactual way, if we gradually weaken the features until the diagnosis changes dramatically, then this feature can be considered as a supporting fact.
Based on this consensus, we propose a counterfactual multi-granularity graph supporting facts extraction (CMGE) method for the irregular EMR in this paper. First, we model the EMR as a hierarchical graph structure, which contains sentences, clauses, and entities. Specifically, sentences are used to model the temporal relationship, clauses provide a complete descriptive explanation, and entities provide symptom support as others. On this basis, we use a graph attention network to aggregate all information from different granularities. Then, we can do a counterfactual intervention to obtain the causal relation between feature and diagnosis. Specifically, we train a learnable soft-mask matrix to mask the feature of nodes or edges in the graph while keeping the diagnosis unchanged, and the remaining features are the supporting facts of the diagnosis. Counterfactual reasoning on the graph requires enhancing the medical features contained in the text of different granularity, so we use clustering labels<sup>4</sup> to cluster clauses and entities. The experimental results demonstrate the effectiveness of our method. The contributions of this paper are summarized as follows:
- We propose a multi-granularity structured
Figure 2: This figure shows the hierarchical connection structure between multi-grained nodes. The black edges in the graph represent the tree structure connection between the four types of nodes in the EMR. For the red edges, the left part shows the connection between the clause nodes and the graph aggregate nodes, and the right part shows the fully connected form between clause nodes.
modeling method based on the hierarchical graph network that decomposes the EMR into sentences, clauses, and entities, and use clustering labels to enhance the expression of medical features.
- We adapt counterfactual intervention to extract critical supporting facts from the EMR during diagnosis. Importantly, our method is disease-independent and does not require a precise external medical knowledge base, so that it is suitable for a wide range of applications.
- The evaluation conducted on the real EMR dataset shows that our method can correctly diagnose the types of lymphedema. Keyword coverage and human evaluation show that the counterfactual reasoning method has better extraction accuracy and robustness compared to two existing methods reimplemented by ourselves.
Given an irregular EMR in the form of free text  $X = [x_{1}, x_{2}, \dots, x_{L}]$  with  $L$  words, the task for us is to extract supporting facts that can be used to explain the diagnosis result without relying on external knowledge while performing diagnosis. The supporting facts can be entities or clauses of text.
2.1 Multi-Granularity Graph Construction
The medical features in the EMR are sparse and medical entities are insufficient to provide sufficient explanation for diagnosis. Therefore, we do multi-granularity segmentation for EMRs, which
4Notice that this label is disease-free and can be initially labeled without expert knowledge by crowdsourcing annotation.
Figure 3: An overview of counterfactual multi-granularity graph supporting facts extraction network. To show the soft-mask process clearly, we assume that the features of both nodes and edges in the graph are 1, and  $\odot$  denotes element-wise multiplication between graph features and mask matrix. All the edges in the graph are bidirectional. For clear reading, we only mark the monodirectional mask value for the bidirectional edges in the Edge-Mask.
enhances the symptom features of entities and explanation of diagnosis, while maintaining the integrity of the text. An EMR can be divided by periods into sentences, which can be further divided into clauses by commas or semicolons as a more granular segmentation. In order to keep the symptom features of entities, we do Named Entity Recognition<sup>5</sup> and number extraction for each clause<sup>6</sup>. In addition, we add two general nodes representing the gender and age of the patient respectively.
After segmentation, as shown in Figure 2, we can build a hierarchical tree structure. The nodes at each level represent the text of sentences, clauses, and entities respectively. Specifically, for each EMR, we connect the two general nodes, sentence nodes sequentially. Then, we connect the clause node to the sentence node to which it belongs and the entity nodes disassembled from it. In particular, a fully-connected relationship is established between all the clause nodes, which overcomes the defect that Graph Attention Network (GAT) can only aggregate the information from adjacent nodes when the network is shallow and expands the receptive field of each sub-sentence node to the whole EMR. Then, all clause nodes are connected to an aggregate node which is used to do the diagnosis. All the edges in the graph are bidirectional to make the information between nodes flow better.
2.2 Clustering labels
In the original EMR, all tokens have the same weight, so noisy text will degrade the performance of diagnosis and explanation. To improve the accuracy of symptom presentation, clustering-labels are used to cluster clauses and entities into corresponding medical classifications. Specifically, the clause is divided into 33 classes and the entity is divide into 10 classes, which is a scientific classification method in medicine derived from the textbook "Diagnostics" (Xuehong Wan, 2013). These labels are disease-free and can be labeled without expert knowledge by crowdsourcing annotation. We manually annotated the corresponding labels for the entire dataset on our own platform. And we have trained a BERT (Devlin et al., 2019) based text classifier on  $30\%$  of the data, which can achieve the annotation accuracy of  $80.76\%$  on clauses and  $97.13\%$  on entities on the remaining data. This shows that our method can easily annotate large-scale data. With these labels, we can gather the same types of features together in the feature space, thereby enhancing the model's overall attention to important types of features. Please refer to Appendix B.2 for more details.
2.3 Input Encoder
After building the multi-granularity graph for a medical record, each node in the graph contains a sequence  $X_{node} = [x_1, x_2, \dots, x_n]$  with  $n$  words, which is tokenized by the tokenizer of BERT (Devlin et al., 2019). In order to maintain the con
$^{5}$ https://github.com/daiyizheng123/Bert-BiLSTM-CRF-pytorch
<sup>6</sup>We recommend Stanza (Qi et al., 2020; Zhang et al., 2020) for English EMR. https://github.com/stanfordlp/stanza
sistency of the results of different granularity encoding, we use one bi-directional RNN (Schuster and Paliwal, 1997) with GRU (Cho et al., 2014) to cover the sequence of sentences, clauses, entities and general information into hidden state sequence respectively  $\pmb{H}_{m} = (\pmb{h}_{1},\pmb{h}_{2},\dots ,\pmb{h}_{n})$  ..
$$
\boldsymbol {h} _ {t} = \boldsymbol {B i G R U} (\boldsymbol {h} _ {t - 1}, \boldsymbol {e} (x _ {t})) \tag {1}
$$
where  $h_t$  is the hidden state of the t-th token and  $e(x_t)$  is the embedding vectors with random initialization of  $x_i$ . Finally, we use the last hidden state of i-th text sequence as the feature  $H_i$  of node  $i$ .
2.4 Graph Reasoning
Once we get the feature of the node, we use the Graph Attention Network (GAT) (Velickovic et al., 2018) to aggregate the information between different granularity. GAT can obtain the correlation score between nodes based on the attention mechanism, which is the key to the interpretability of our model. Specifically, GAT takes all the node features as input and calculates the attention coefficients  $\alpha_{ij}$  by
$$
e _ {i j} = \text {L e a k y R e L U} \left(\boldsymbol {a} ^ {T} \left(\left[ \boldsymbol {W H} _ {i}; \boldsymbol {W H} _ {j} \right]\right)\right) \tag {2}
$$
$$
\alpha_ {i j} = \frac {\exp \left(e _ {i j}\right)}{\sum_ {k \in \mathcal {N} _ {i}} \exp \left(e _ {i k}\right)} \tag {3}
$$
where  $H_{i}$  is the feature of node  $i$ ,  $W \in \mathbb{R}^{d \times d}$  is a learnable weight matrix for the linear projection,  $a \in \mathbb{R}^{2d}$  is a learnable weight vector used to transform the adjacent node feature representations to the edge score  $e_{ij}$  between the  $i$ -th and  $j$ -th nodes. Equation (4) means to do a softmax normalization between all the edge attention scores on the edges connected to node  $i$ . Then, we update the feature of each node by
$$
\boldsymbol {H} _ {i} ^ {\prime} = \text {L e a k y R e L U} \left(\sum_ {j \in \mathcal {N} _ {i}} \alpha_ {i j} \boldsymbol {W} \boldsymbol {H} _ {j}\right) \tag {4}
$$
After graph reasoning, the representation  $\pmb{H}$  of each node has been updated with the granular information aggregated from adjacent nodes and can be used for subsequent tasks.
2.5 Multi-task Prediction
After obtaining the updated node features, we use them in three subtasks: (i) graph classification for automatic diagnosis; (ii) sub-sentence classification for clustering; and (iii) entity classification for clustering.
Taking entity node classification as an example, for each entity node, we use a two-layer MLP with the ReLU activation function to calculate the probability. For an entity node  $i$ , we can get
$$
P _ {e n t i t y, i} = M L P _ {e n t i t y} \left(\boldsymbol {E} _ {i}\right) \tag {5}
$$
By the same way, we can obtain the probability  $P_{graph}, P_{clause}, P_{entity}$ . The same as the common multi-task learning, we joint all the losses together as:
$$
\mathcal {L} _ {\text {j o i n t}} = \lambda_ {1} \mathcal {L} _ {\text {g r a p h}} + \lambda_ {2} \mathcal {L} _ {\text {c l a u s e}} + \lambda_ {3} \mathcal {L} _ {\text {e n t i t y}} \tag {6}
$$
where  $\lambda_{1},\lambda_{2}$  and  $\lambda_{3}$  are hyper-parameters, and all the loss are calculated by cross-entropy loss.
2.6 Counterfactual Reasoning on Graph
Providing supporting information while making the diagnosis is the key to applying Artificial Intelligence into the medical field. Inspired by (Ying et al., 2019), we add node-mask or edge-mask into GAT to obtain the counterfactual result after the training and eliminate the noise nodes while keeping the diagnostic results unchanged.
For edge-mask, we introduce a learnable matrix  $M$  with the same form as the adjacency matrix of the medical record graph. Each element  $m_{ij}$  in the matrix represents the degree of mask for message aggregation from node  $i$  to node  $j$  in the graph. With this method, the calculation of attention coefficients in the GAT has been changed to
$$
\alpha_ {i j} = \frac {\exp \left(e _ {i j} m _ {i j}\right)}{\sum_ {k \in \mathcal {N} _ {i}} \exp \left(e _ {i k} m _ {i k}\right)} \tag {7}
$$
And for node-mask, similarly, we introduce a learnable parameter  $\beta_{i}$  for each node  $i$  in the graph. The parameter represents the degree of mask for the feature in the node. After node-mask, the calculation of  $e_{ij}$  and  $H_{i}^{\prime}$  has been changed to
$$
e _ {i j} = \text {L e a k y R e L U} \left(a ^ {T} \left(\left[ \beta_ {i} \boldsymbol {W} \boldsymbol {H} _ {i}; \beta_ {j} \boldsymbol {W} \boldsymbol {H} _ {j} \right]\right)\right) \tag {8}
$$
$$
\boldsymbol {H} _ {i} ^ {\prime} = \text {L e a k y R e L U} \left(\sum_ {j \in \mathcal {N} _ {i}} \alpha_ {i j} \beta_ {j} \boldsymbol {W} \boldsymbol {H} _ {j}\right) \tag {9}
$$
In the training of counterfactual reasoning, we jointly optimize three loss functions to obtain accurate counterfactual results. To ensure that the model can make a correct diagnosis after the counterfactual intervention, we use the original model
type	train	test
Secondary Lymphedema	448	36
Primary Lymphedema	185	22
Chylous Reflux Lymphedema	19	21
Others	248	21
All	900	100
to obtain the fact result  $D_{i}$  and maximizes the probability of selecting the correct diagnosis in counterfactual reasoning. Besides, we minimize the sum of all elements in the mask matrix to ensure that all noise nodes are filtered as much as possible. Since there is an exponential level possibility of counterfactual intervention on the model through the node-mask or edge-mask, we minimize the information entropy of the mask matrix regarding which node to select to reduce the uncertainty of the result. Finally, the loss of counterfactual reasoning is as follows:
$$
\begin{array}{l} \mathcal {L} _ {c} = - \lambda_ {4} \log P (D = D _ {i}) + \lambda_ {5} \operatorname {s u m} (\boldsymbol {M}) \\ - \lambda_ {6} \frac {1}{N} \sum_ {m _ {i} \in M} m _ {i} \log (m _ {i}) \tag {10} \\ - \lambda_ {6} \frac {1}{N} \sum_ {m _ {i} \in M} (1 - m _ {i}) l o g (1 - m _ {i}) \\ \end{array}
$$
where  $\lambda_4, \lambda_5$  and  $\lambda_6$  are hyper-parameters,  $N$  is the number of elements in the mask matrix  $M$ , and all the elements in  $M$  are mapped to the  $[0,1]$  by sigmoid function. For node-mask, the training is similar.
After counterfactual reasoning, we extract the nodes or edges (each edge represents the two nodes connected) represented by the top-k elements in the mask matrix as supporting facts.
3.1 Experimental Setting
Based on the cooperation with the hospitals, we conducted experiments with real EMR data. We selected the EMRs from the department of lymphedema and diagnose the disease of primary lymphedema (原发性淋巴水肿), secondary lymphedema (继发性淋巴水肿), chylous reflux lymphedema (乳糜返流性淋巴水肿) and others (其他). The reasons for us to choose this department are as follows: (I) Lymphedema is a sub-discipline in medicine, so the researches on it, whether in Medicine or Artificial Intelligence, is still limited.
For example, ICD10 can not provide full medical supporting. (II) The pathogenesis and treatment methods of different types of lymphedema vary greatly, but their outward manifestations are similar. Therefore, there is an urgent need for a simple method of earlier diagnosis system of lymphedema. (III) Specialist doctors pay more attention to the diagnosis in sub-discipline disease and do not concern with the large-scale rough diagnosis.
Formally, there are 1000 EMRs used in our experiment, of which 900 are used for training and 100 are used for testing. The statistics of four types of diseases are shown in Table 1. The average length of all EMRs is 345 words in Chinese. And our model is implemented based on PyTorch (Paszke et al., 2019), and use Adam (Kingma and Ba, 2015) optimizer for training. Please refer to Appendix B.1 for datasets details and Appendix A.1 for implementation details.
3.2 Baseline
We designed two representative models to compare the ability to extract medical support facts under similar task conditions based on attention and variational inference:
Self-Attention This method represents most of the existing approaches and provides explanations through attention similarity. We use BiGRU to encode the EMR. With the sequence embedding, following (Choi et al., 2016), we use average pooling to obtain the overall representation for automatic diagnosis. For supporting fact extraction, following (Mullenbach et al., 2018), we calculate the self-attention weight of each token, and design a sliding window method to obtain the average attention scores of fixed length spans, among which having high scores are taken as the supporting facts.
PostKS This is another method based on variational inference we've designed in addition to attention. Inspired by the dialogue knowledge selection model PostKS (Lian et al., 2019), we convert the pivotal information extraction into a clause selection problem. This method uses the text result of the diagnosis(as shown in Figure 1) to calculate the correlation with the clause as posterior distribution through the attention mechanism, and then uses self-attention and average pooling between clauses to obtain correlation score as the prior distribution. During training, based on variational inference, the model uses posterior information to guide prior selection, so that makes the prior distribution and the
Model	Diagnosis	Clause	Entity
P/%	R/%	F1/%	P/%	R/%	F1/%	P/%	R/%	F1/%
Self-Attention	94.95	95.00	94.97	-	-	-	-	-	-
PostKS	95.13	97.00	96.06	-	-	-	-	-	-
CMGE-c-e	96.17	96.00	96.08	1.91	3.72	2.52	14.36	9.05	11.10
CMGE-e	97.40	96.00	96.69	81.26	81.80	81.53	15.22	32.18	20.67
CMGE-c	97.19	97.11	97.15	25.75	1.55	2.92	95.33	95.12	95.22
CMGE	99.04	99.00	99.02	82.49	82.53	82.51	96.43	96.38	96.40
Table 2: The first two lines are the diagnostic performance of the compared model and the last line is ours. The middle three rows are ablation experiments.  $\mathrm{CMGE}_{-c - e}$  represents the model only use diagnosis label,  $\mathrm{CMGE}_{-e}$  represents the model without entity labels, and  $\mathrm{CMGE}_{-c}$  represents the model without clause labels.
posterior distribution consistent. Finally, during inference, we select the clauses with high prior attention scores among clauses as supporting facts. Please refer to Appendix A.2 for more details.
To measure the performance of our pivotal information extraction module, we built a simple diagnostic criterion from (Levine, 2017), which is a complete diagnosis and treatment guide for lymphedema written by medical experts. Based on this diagnosis criteria, we used a combination of automatic evaluation and human evaluation.
Automatic Evaluation The precision, recall, and F1 are used as the metrics to measure the diagnostic accuracy of the model, which is the basis for the practical application. Specifically, several key-phrases for the three types of lymphedema are manually identified respectively to represent diagnostic features, and they are the re-descriptions of diagnostic criteria in the guide using phrases from EMRs. We use hit@1/3/5 (Bordes et al., 2013) to measure the coverage rate of the extracted results to the key-phrases. These metrics represent whether one of the diagnostic features is included in the top-1/3/5 extracted results. Please refer to Appendix B.3 for more details.
Human Evaluation Since some of the implicit medical features cannot be covered by key-phrases, human evaluation is necessary. We used each model to extract the top 3 supporting facts respectively for all 100 EMR samples in the testset, and randomly shuffled the order of the results. Then we invited 3 evaluators with medical backgrounds and having read the guide to determine whether the results conform to medical knowledge. We focus on the comprehensiveness and trustworthiness of each model. Comprehensiveness is used to mea
sure whether the model can provide more medical features, and trustworthiness is used to measure whether the extraction results are helpful for diagnosis. For each item, the evaluator is asked to score in  $0 \sim 2$ . The final indicator is the average of the three evaluators.
4.1 Diagnostic Result