SlowGuess/ACL_2020_OCR huggingface dataset:
Pengfei Cao $^{1,2}$ , Yubo Chen $^{1,2}$ , Kang Liu $^{1,2}$ , Jun Zhao $^{1,2}$ , Shengping Liu $^{3}$  and Weifeng Chong $^{3}$
$^{1}$  National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China
$^{2}$  School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100049, China
<sup>3</sup> Beijing Unisound Information Technology Co., Ltd, Beijing, 100028, China {pengfei.cao, yubo.chen, kliu, jzhao} @ nlpr.ia.ac.cn, {liushengping, chongweifeng} @ unisound.com
The International Classification of Diseases (ICD) provides a standardized way for classifying diseases, which endows each disease with a unique code. ICD coding aims to assign proper ICD codes to a medical record. Since manual coding is very laborious and prone to errors, many methods have been proposed for the automatic ICD coding task. However, most of existing methods independently predict each code, ignoring two important characteristics: Code Hierarchy and Code Co-occurrence. In this paper, we propose a Hyperbolic and Co-graph Representation method (HyperCore) to address the above problem. Specifically, we propose a hyperbolic representation method to leverage the code hierarchy. Moreover, we propose a graph convolutional network to utilize the code co-occurrence. Experimental results on two widely used datasets demonstrate that our proposed model outperforms previous state-of-the-art methods.
The International Classification of Diseases (ICD) is a healthcare classification system supported by the World Health Organization, which provides a unique code for each disease, symptom, sign and so on. ICD codes have been widely used for analyzing clinical data and monitoring health issues (Choi et al., 2016; Avati et al., 2018). Due to the importance of ICD codes, ICD coding – which assigns proper ICD codes to a medical record – has drawn much attention. The task of ICD coding is usually undertaken by professional coders according to doctors' diagnosis descriptions in the form of free texts. However, manual coding is very expensive, time-consuming and error-prone.
Figure 1: An example of automatic ICD coding task. The input and output of the automatic ICD coding model are clinical text and predicted ICD codes, respectively. For better understanding, we add the corresponding disease name for each code.
The cost incurred by coding errors and the financial investment spent on improving coding quality are estimated to be \(25 billion per year in the US (Lang, 2007). Two main reasons can account for this. First, only the people who have medical expert knowledge and specialized ICD coding skills can handle the task. However, it is hard to train such an eligible ICD coder. Second, it is difficult to correctly assign proper codes to the input document even for professional coders, because one document can be assigned multiple ICD codes and the number of codes in the taxonomy of ICD is large. For example, there are over 15,000 and 60,000 codes respectively in the ninth version (ICD-9) and the tenth version (ICD-10) of ICD taxonomies.
To reduce human labor and coding errors, many methods have been carefully designed for automatic ICD coding (Perotte et al., 2013; Mullenbach et al., 2018). For example in Figure 1, given the clinical text of a patient, the ICD coding model needs to automatically predict the corresponding ICD codes. The automatic ICD coding task can be modeled as a multi-label classification task since each clinical text is usually accompanied by mul
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3105-3114 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 2: An example of ICD-9 descriptors and the derived hierarchical structure.
tiple codes. Most of the previous methods handle each code in isolation and convert the multi-label problem into a set of binary classification problems to predict whether each code of interest presents or not (Mullenbach et al., 2018; Rios and Kavuluru, 2018). Though effective, they ignore two important characteristics: Code Hierarchy and Code Co-occurrence, which can be leveraged to improve coding accuracy. In the following, we will introduce the two characteristics and the reasons why they are critical for the automatic ICD coding.
Code Hierarchy: Based on ICD taxonomy, ICD codes are organized under a tree-like hierarchical structure as shown in Figure 2, which indicates the parent-child and sibling relations between codes. In the hierarchical structure, the upper level nodes represent more generic disease categories and the lower level nodes represent more specific diseases. The code hierarchy can capture the mutual exclusion of some codes. If code X and Y are both children of Z (i.e., X and Y are the siblings), it is unlikely to simultaneously assign X and Y to a patient in general (Xie and Xing, 2018). For example in Figure 2, if code "464.00 (acute laryngitis without mention of obstruction)" is assigned to a patient, it is unlikely to assign the code "464.01 (acute laryngitis with obstruction)" to the patient at the same time. If automatic ICD coding models ignore such a characteristic, they are prone to giving inconsistent predictions. Thus, a challenging problem is how to model the code hierarchy and use it to capture the mutual exclusion of codes.
Code Co-occurrence: Since some diseases are concurrent or have a causal relationship with each other, their codes usually co-occur in the clinical text, such as "997.91 (hypertension)" and "429.9 (heart disease)". In this paper, we call such characteristic code co-occurrence which can capture the correlations of codes. The code co-occurrence can be utilized to correctly predict some codes which are difficult to predict by only using the clinical text
itself. For example in Figure 1, the code of "acute respiratory failure" can be easily inferred via capturing apparent clues (i.e., the green bold words) from the text. Although there are also a few clues to infer the code of "acidosis", they are very obscure, let alone predict the code of "acidosis" by only using these obscure clues. Fortunately, there is a strong association between these two diseases: one of the main causes of "acidosis" is "acute respiratory failure". This prior knowledge can be captured via the fact that the codes of the two diseases usually co-occur in clinical texts. By considering the correlation, the automatic ICD coding model can better exploit obscure clues to predict the code of "acidosis". Therefore, another problem is how to leverage code co-occurrence for ICD coding.
In this paper, we propose a novel method termed as Hyperbolic and Co-graph Representation method (HyperCore) to address above problems. Since the tree-likeness properties of the hyperbolic space make it more suitable for representing symbolic data with hierarchical structures than the Euclidean space (Nickel and Kiela, 2017), we propose a hyperbolic representation learning method to learn the Code Hierarchy. Meanwhile, the graph has been proved effective in modeling data correlation and the graph convolutional network (GCN) enables to efficiently learn node representation (Kipf and Welling, 2016). Thus, we devise a code co-occurrence graph (co-graph) for capturing Code Co-occurrence and exploit the GCN to learn the code representation in the co-graph.
The contributions of this paper are threefold. Firstly, to our best knowledge, this is the first work to propose a hyperbolic representation method to leverage the code hierarchy for automatic ICD coding. Secondly, this is also the first work to utilize a GCN to exploit code co-occurrence correlation for automatic ICD coding. Thirdly, experiments on two widely used automatic ICD coding datasets show that our proposed model outperforms previous state-of-the-art methods.
Automatic ICD Coding. Automatic ICD coding is a challenging and important task in the medical informatics community, which has been studied with traditional machine learning methods (Larkey and Croft, 1996; Perotte et al., 2013) and neural network methods (Koopman et al., 2015; Rios and Kavuluru, 2018; Yu et al., 2019). Given discharge
Figure 3: The architecture of Hyperbolic and Co-graph Representation method (HyperCore). In the Poincaré ball  $\mathcal{B}^n$ , we show the embedded code hierarchy (i.e., tree-like hierarchical structure). The dots  $l_i$  ( $i = 1,2,3$ ) on the tree-like hierarchical structure and triangles  $m_i$  ( $i = 1,2,3$ ) in the Poincaré ball denote hyperbolic code embeddings and hyperbolic document representations, respectively.
summaries, Perotte et al. (2013) propose a hierarchical SVM model to predict ICD codes. Recently, neural network methods have been introduced to the task. Mullenbach et al. (2018) propose an attention based convolutional neural network (CNN) model to capture important information for each code. Xie and Xing (2018) adopt tree long short-term memory (LSTM) to utilize code descriptions. Though effective, they ignore the code hierarchy and code co-occurrence.
Hyperbolic Representation. Hyperbolic space has been applied to modeling complex networks (Krioukov et al., 2010). Recent research on representation learning demonstrates that the hyperbolic space is more suitable for representing symbolic data with hierarchical structures than the Euclidean space (Nickel and Kiela, 2017, 2018; Hamann, 2018). In the field of natural language processing (NLP), the hyperbolic representation has been successfully applied to question answering (Tay et al., 2018), machine translation (Gulcehre et al., 2018) and sentence representation (Dhingra et al., 2018). To our knowledge, this is the first work to apply hyperbolic representation method to the automatic ICD coding task.
Graph Convolutional Networks. GCN (Kipf and Welling, 2016) is a powerful neural network, which operates on graph data. It yields substantial improvements over various NLP tasks such as semantic role labeling (Marcheggiani and Titov, 2017), multi-document summarization (Yasunaga et al., 2017) and machine translation (Bastings et al., 2017). Velicković et al. (2017) propose graph atten
tion networks (GAT) to summarize neighborhood features by using masked self-attentional layers. We are the first to capture the code co-occurrence characteristic via the GCN for the automatic ICD coding task.
We propose a hyperbolic and co-graph representation (HyperCore) model for automatic ICD coding. Firstly, to capture the code hierarchy, we learn the code hyperbolic representations and measure the similarities between document and codes in the hyperbolic space. Secondly, to exploit code cooccurrence, we exploit the GCN to learn code cooccurrence representations and use them as query vectors to obtain code-aware document representations. Finally, the document-code similarity scores and code-aware document representations are then aggregated to predict the codes. Figure 3 shows the overall architecture of our proposed model.
3.1 Convolution Neural Network Encoder
We first map each word into a low dimensional word embedding space. The document can be denoted as  $\mathbf{X} = \{\pmb{x}_1, \pmb{x}_2, \dots, \pmb{x}_N\}$ , where  $N$  is the length of the document. Then, we exploit the CNN to encode the clinical text due to its high computational efficiency:
$$
\boldsymbol {h} _ {i} = \tanh  \left(\boldsymbol {W} _ {c} * \boldsymbol {x} _ {i: i + k - 1} + \boldsymbol {b} _ {c}\right) \tag {1}
$$
where  $W_{c}$  is the convolutional filter.  $b_{c}$  is the bias.  $k$  is the filter size. * is the convolution operator.
3.2 Code-wise Attention
After encoding by CNN, we obtain the document representation  $H = \{h_1, h_2, \ldots, h_N\}$ . Since we need to assign multiple codes for each document and different codes may focus on different sections of the document, we employ code-wise attention to learn relevant document representations for each code. We first generate the code vector for each code via averaging the word embeddings of its descriptor:
$$
\boldsymbol {v} _ {i} = \frac {1}{N _ {d}} \sum_ {j = 1} ^ {N _ {d}} \boldsymbol {w} _ {j}, \quad i = 1, \dots , L \tag {2}
$$
where  $\pmb{v}_i$  is the code vector,  $N_d$  is the length of the descriptor,  $\pmb{w}_j$  is the embedding of  $j$ -th word in the descriptor, and  $L$  is the total number of codes in the dataset (Jouhet et al., 2012; Johnson et al., 2016). The code vectors set is  $\mathbf{V} = \{\pmb{v}_1, \pmb{v}_2, \dots, \pmb{v}_L\}$ .
Then, we generate the code-wise attention vector via matrix-vector product:
$$
\boldsymbol {\alpha} _ {i} = \operatorname {s o f t m a x} \left(\boldsymbol {H} ^ {T} \boldsymbol {v} _ {i}\right) \tag {3}
$$
Finally, we use the document representation  $H$  and attention vector  $\alpha_{i}$  to generate the code-aware document representation:
$$
\boldsymbol {c} _ {i} = \boldsymbol {H} \boldsymbol {\alpha} _ {i} \tag {4}
$$
We concatenate the  $c_{i}$  ( $i = 1, \dots, L$ ) to obtain the code-aware document representation, denoted as  $C = \{c_{1}, c_{2}, \dots, c_{L}\} \in \mathbb{R}^{d_{c} \times L}$ .
3.3 Document-Code Similarities in Hyperbolic Space
To capture the code hierarchy, we learn the code hyperbolic representations and measure the similarities between document and codes in the hyperbolic space. In this section, we propose a hyperbolic code embedder to obtain code hyperbolic representations, and we also propose a hyperbolic document projector to project the document representations from Euclidean space to hyperbolic space. We then compute the similarities between the document and codes in the hyperbolic space.
3.3.1 Hyperbolic Geometry
Hyperbolic geometry is a non-Euclidean geometry which studies spaces of constant negative curvature. Our approach is based on the Poincaré ball model (Nickel and Kiela, 2017), which is a particular model of hyperbolic space and is well-suited for gradient-based optimization. In particular, let  $\mathcal{B}^n = \{\pmb{x} \in \mathbb{R}^n \mid ||\pmb{x}|| < 1\}$  be the open
$n$ -dimensional unit ball, where  $||\cdot||$  denotes the Euclidean norm. The Poincaré ball  $(\mathcal{B}^n,g_x)$  is defined by the Riemannian manifold, i.e., the open unit ball equipped with the Riemannian metric tensor:
$$
g _ {\boldsymbol {x}} = \left(\frac {2}{1 - \left\| \boldsymbol {x} \right\| ^ {2}}\right) ^ {2} g ^ {E} \tag {5}
$$
where  $\pmb{x} \in \mathcal{B}^n$ .  $g^{E}$  denotes the Euclidean metric tensor. Furthermore, the distance between two points  $\pmb{u}, \pmb{v} \in \mathcal{B}^n$  is given as:
$$
d (\boldsymbol {u}, \boldsymbol {v}) = \operatorname {a r c o s h} (1 + 2 \frac {\left\| \boldsymbol {u} - \boldsymbol {v} \right\| ^ {2}}{\left(1 - \left\| \boldsymbol {u} \right\| ^ {2}\right) \left(1 - \left\| \boldsymbol {v} \right\| ^ {2}\right)}) \tag {6}
$$
where  $\operatorname{arccosh}(x) = \ln (x + \sqrt{(x^2 - 1)})$ . If we consider the origin  $O$  and two points  $\mathbf{u}, \mathbf{v}$ , when the two points moving towards the outside of the Poincaré ball (i.e.,  $||\mathbf{u}||, ||\mathbf{v}|| \to 1$ ), the distance  $d(\mathbf{u}, \mathbf{v})$  tends to  $d(\mathbf{u}, \mathbf{O}) + d(\mathbf{O}, \mathbf{v})$ . That is, the path between the two points converges to a path through the origin, which can be seen as a tree-like hierarchical structure.
3.3.2 Hyperbolic Code Embedder
The tree-likeness of the hyperbolic space makes it natural to embed hierarchical structures. By embedding code hierarchy in the Poincaré ball, the top codes are placed near the origin and bottom codes are near the boundary. The embedding norm represents depth in the hierarchy, and the distance between embeddings represents the similarity. Let  $\mathcal{D} = \{(l_p,l_q)\}$  be the set of parent-child relations between code pairs.  $\Theta = \{\pmb {\theta}_i\}_{i = 1}^T,\pmb {\theta}_i\in \mathcal{B}^{d_p}$  is the corresponding code embedding set, where  $T$  is the number of all ICD codes. In order to enforce related codes to be closer than unrelated codes, we minimize the following loss function to get the code hyperbolic representations when  $\| \pmb {\theta}_i\| < 1(i = 1,\dots ,L)$ :
$$
\mathcal {J} (\boldsymbol {\Theta}) = - \sum_ {(l _ {p}, l _ {q}) \in \mathcal {D}} \log \frac {\exp (- d \left(\boldsymbol {\theta} _ {p} , \boldsymbol {\theta} _ {q}\right))}{\sum_ {l _ {q ^ {\prime}} \in \mathcal {N} (l _ {p})} \exp (- d \left(\boldsymbol {\theta} _ {p} , \boldsymbol {\theta} _ {q ^ {\prime}}\right))} \tag {7}
$$
where  $\mathcal{N}(l_p) = \{l_{q'}|(l_p,l_{q'})\notin \mathcal{D}\} \cup \{l_p\}$  is the set of negative samples. The hyperbolic code representations in our work are denoted as  $\Theta_L = \{\pmb {\theta}_i\}_{i = 1}^L$ $d(\cdot)$  is the distance defined as Equation (6).
3.3.3 Hyperbolic Document Projector
To compute the similarities between document and codes in hyperbolic space, the code-aware document representations  $C = \{c_1, c_2, \ldots, c_L\}$  need
to be projected into the hyperbolic space. We exploit the re-parameterization technique (Dhingra et al., 2018; López et al., 2019) to implement it, which involves computing a direction vector  $\boldsymbol{r}$  and a norm magnitude  $\eta$ . We use the  $c_{i}$  as an example to illustrate the procedure:
$$
\overline {{\boldsymbol {r}}} _ {i} = \Phi_ {\text {d i r}} (\boldsymbol {c} _ {i}), \quad \boldsymbol {r} _ {i} = \frac {\overline {{\boldsymbol {r}}} _ {i}}{| | \overline {{\boldsymbol {r}}} _ {i} | |} \tag {8}
$$
$$
\overline {{\eta}} _ {i} = \varPhi_ {n o r m} (\pmb {c} _ {i}), \quad \eta_ {i} = \sigma (\overline {{\eta}} _ {i})
$$
where  $\varPhi_{dir}:\mathbb{R}^{d_c}\to \mathbb{R}^{d_p}$  is the direction function. We parameterize it as a multi-layer perceptron (MLP).  $\varPhi_{norm}:\mathbb{R}^{d_c}\rightarrow \mathbb{R}$  is the norm magnitude function. We use a linear layer to implement it.  $\sigma$  is the sigmoid function to ensure the resulting norm  $\eta_i\in (0,1)$ . The re-parameterized document representation is defined as  $m_{i} = \eta_{i}\pmb{r}_{i}$ , which lies in hyperbolic space  $\mathcal{B}^{d_p}$ .
The re-parameterization technique enables to project the code-aware document representation into the Poincaré ball, which enables the avoidance of the stochastic Riemannian optimization method (Bonnabel, 2013) to learn the parameters in the hyperbolic space. Instead, we can exploit the deep learning optimization method to update the parameters in the entire model.
3.3.4 Compute Document-Code Similarity
Since there doesn't exist a clear hyperbolic innerproduct, the cosine similarity is not appropriate to be the metric. In our work, we adopt the hyperbolic distance function to model the relationships between the document and codes. Since the hyperbolic document representation for each code has been obtained, we just need to compute the similarity with the corresponding hyperbolic code embedding:
$$
\begin{array}{l} \text {s c o r e} _ {i} = d (\boldsymbol {m} _ {i}, \boldsymbol {\theta} _ {i}) \\ \text {a l p h a} \end{array} \tag {9}
$$
$$
\boldsymbol {S} = \left[ \begin{array}{c} s c o r e _ {1}; s c o r e _ {2}; \dots ; s c o r e _ {L} \end{array} \right]
$$
where  $S \in \mathbb{R}^L$  is the document-code similarity score.  $[;]$  is the concatenation operation.  $d(\cdot)$  is the distance function defined as Equation (6).
3.4 Code-aware Document Representations via Graph Convolutional Network
To exploit code co-occurrence, we exploit the graph to model code co-occurrence correlation, and then we use the GCN to learn code cooccurrence representations. In this section, we first construct the co-graph according to the statistics of the code co-occurrence in the training set, and then we exploit
the GCN to encode the code co-occurrence correlation.
3.4.1 Code Co-graph Construction
Given a graph with  $L$  nodes, we can represent the graph using a  $L \times L$  adjacency matrix  $\mathbf{A}$ . To capture the co-occurrence correlations between codes, we build the code co-occurrence graph (co-graph), which utilizes the code co-occurrence matrix as the adjacency matrix. If the  $i$ -th code and the  $j$ -th code co-occur in the clinical text, there is an edge between them. Intuitively, if the  $i$ -th code co-appears with the  $j$ -th code more often than the  $k$ -th code, the probabilities of the  $i$ -th code and the  $j$ -th code should have stronger dependencies. Therefore, in our work, we use the co-appearing times between two codes as the connection weights in the adjacency matrix, which can represent the prior knowledge. For example, if the  $i$ -th code co-appears  $n$  times with the  $j$ -th code, we set  $\mathbf{A}_{ij} = n$ .
3.4.2 Code Co-occurrence Encoding via GCN
The inputs of GCN are initial representations of codes  $V$  which are obtained via Equation (2) and the adjacency matrix  $\mathbf{A}$ . We use the standard convolution computation (Kipf and Welling, 2016) to encode code co-occurrence:
$$
\boldsymbol {H} ^ {(l + 1)} = \rho \left(\tilde {\boldsymbol {D}} ^ {- \frac {1}{2}} \tilde {\boldsymbol {A}} \tilde {\boldsymbol {D}} ^ {- \frac {1}{2}} \boldsymbol {H} ^ {(l)} \boldsymbol {W} ^ {(l)}\right) \tag {10}
$$
where  $\tilde{\pmb{A}} = \pmb {A} + \pmb{I}$ $I$  is the identity matrix,  $\tilde{D}_{ii} = \sum_{j}\tilde{A}_{ij},H^{(l)}\in \mathbb{R}^{L\times d_c}$  and  $\pmb{H}^{(0)} = \pmb{V}$ $\rho$  is an activation function (e.g., ReLU). After co-occurrence correlation encoding via GCN, the code representations enable to capture the code co-occurrence correlations. Then, we use the codewise attention to obtain code-aware document representations, denoted as  $\pmb {D} = \{\pmb {d}_1,\pmb {d}_2,\dots ,\pmb {d}_L\} ^1$
3.5 Aggregation Layer
After capturing the code hierarchy and code co-occurrence, we use an aggregation layer to fuse document-code similarity scores  $S$  and code-aware document representations  $D$  for enhancing representation with each other:
$$
\boldsymbol {U} = \lambda \boldsymbol {W} _ {s} \boldsymbol {S} + \boldsymbol {D} ^ {T} \boldsymbol {W} _ {d} \tag {11}
$$
where  $\mathbf{W}_s$  and  $\mathbf{W}_d$  are transformation matrixes.  $\mathbf{U} = \{u_1, u_2, \dots, u_L\} \in \mathbb{R}^L$  are final document representations for each code.  $\lambda$  is the hyperparameter.
${}^{1}C$  and  $D$  are both code-aware document representations, but  $D$  captures the code co-occurrence correlations.
Model	MIMIC-III full	MIMIC-III 50
AUC	F1	P@N	AUC	F1	P@5
Macro	Micro	Macro	Micro	8	15	Macro	Micro	Macro	Micro
C-MemNN	-	-	-	-	-	-	0.833	-	-	-	0.420
C-LSTM-ATT	-	-	-	-	-	-	-	0.900	-	0.532	-
CAML	0.895	0.986	0.088	0.539	0.709	0.561	0.875	0.909	0.532	0.614	0.609
DR-CAML	0.897	0.985	0.086	0.529	0.690	0.548	0.884	0.916	0.576	0.633	0.618
HyperCore	0.930	0.989	0.090	0.551	0.722	0.579	0.895	0.929	0.609	0.663	0.632
±0.001	±0.005	±0.003	±0.001	±0.002	±0.001	±0.003	±0.002	±0.001	±0.001	±0.002
The prediction for each code is generated via:
$$
\hat {y} _ {i} = \sigma \left(u _ {i}\right), \quad i = 1, \dots , L \tag {12}
$$
Our model is to be trained using a multi-label binary cross-entropy loss:
$$
\mathcal {L} = \sum_ {i = 1} ^ {L} \left[ - y _ {i} \log \left(\hat {y} _ {i}\right) - \left(1 - y _ {i}\right) \log \left(1 - \hat {y} _ {i}\right) \right] \tag {13}
$$
where  $y_{i}\in \{0,1\}$  is the ground truth for the  $i$  -th code.
We evaluate our proposed model on two widely used datasets, including MIMIC-II (Jouhet et al., 2012) and MIMIC-III (Johnson et al., 2016). Both datasets contain discharge summaries that are tagged by human coders with a set of ICD-9 codes. For MIMIC-III dataset, we use the same experimental setting as previous works (Shi et al., 2017; Mullenbach et al., 2018). The dataset has two common settings: MIMIC-III full and MIMIC-III 50. For MIMIC-III full setting, the setting consists of 8921 codes, 47719, 1631 and 3372 discharge summaries for training, development and testing respectively. For MIMIC-III 50 setting, the setting contains the top 50 most frequent codes, 8067, 1574 and 1730 discharge summaries for training, development and testing respectively. For the MIMIC-II dataset, we use the same splits as previous works (Perotte et al., 2013; Mullenbach et al., 2018), there are 20533 and 2282 clinical notes for training and testing, and 5031 unique ICD-9 codes in the dataset.
4.2 Metrics and Parameter Settings