SlowGuess/ACL_2020_OCR huggingface dataset:
Following previous work (Mullenbach et al., 2018), we use macro-averaged and micro-averaged F1, macro-averaged and micro-averaged AUC (area under the ROC, i.e., receiver operating characteristic curve) and Precision@N (P@N) as the metrics.
Table 1: Comparison of our model and other baselines on the MIMIC-III dataset. We run our model 10 times and each time we use different random seeds for initialization. We report the mean ± standard deviation of each result.
Model	AUC	F1	P@8
Macro	Micro	Macro	Micro
SVM	-	-	-	0.293	-
HA-GRU	-	-	-	0.366	-
CAML	0.820	0.966	0.048	0.442	0.523
DR-CAML	0.826	0.966	0.049	0.457	0.515
HyperCore	0.885	0.971	0.070	0.477	0.537
±0.001	±0.004	±0.002	±0.003	±0.003
Table 2: Experimental results are shown in means  $\pm$  standard deviations on the MIMIC-II dataset.
The  $\mathrm{P@N}$  indicates the proportion of the correctly-predicted labels in the top-N predicted labels.
Hyper-parameters are tuned on the development set by grid search. The word embedding size  $d_{e}$  is 100. The convolution filter size is 10. The size of the filter output is 200. The dropout rate is 0.4. The  $\lambda$  is 0.2. The batch size is 16. Adam (Kingma and Ba, 2014) is used for optimization with an initial learning rate 1e-4. We pre-train the word embeddings on the combination of training sets of MIMIC-II and MIMIC-III datasets by using word2vec toolkit (Mikolov et al., 2013).
SVM: A hierarchical support vector machine (SVM) is proposed by Perotte et al. (2013) to use the hierarchical nature of ICD codes, which is evaluated on the MIMIC-II dataset.
C-MemNN: A condensed memory neural network is proposed by Prakash et al. (2017) to predict ICD codes on the MIMIC-III 50 dataset.
C-LSTM-ATT: A character-aware LSTM based attention model is proposed by Shi et al. (2017). It is also evaluated on the MIMIC-III 50 dataset.
HA-GRU: A hierarchical attention gated recurrent unit model is proposed by Baumel et al. (2018) to predict ICD codes on the MIMIC-II dataset.
CAML & DR-CAML: The convolutional attention network for multi-label classification (CAML) is proposed by Mullenbach et al. (2018). DR-CAML is an extension of CAML which
Models	MIMIC-III full	MIMIC-III 50	MIMIC-II
Macro-F1	Micro-F1	Macro-F1	Micro-F1	Macro-F1	Micro-F1
HyperCore	0.090	0.551	0.609	0.663	0.070	0.477
w/o hyperbolic representation	0.081	0.539	0.576	0.645	0.062	0.464
w/o co-graph representation	0.085	0.541	0.582	0.637	0.055	0.453
w/o hyperbolic and co-graph representation	0.077	0.531	0.570	0.626	0.047	0.439
Table 3: Ablation study by removing the main components, where "w/o" indicates without.
incorporates the code description. They achieve the state-of-the-art performance on the MIMIC-III and MIMIC-II datasets.
4.4 Compared with State-of-the-art Methods
We repeat 10 times training and each time we use different random seeds for initialization. We report the mean  $\pm$  standard deviation of each result. Table 1 and Table 2 show the results on the MIMIC-III and MIMIC-II datasets, respectively. Since some baselines are evaluated either on MIMIC-III or MIMIC-II, the baselines used for the two datasets are different. Overall, we observe that:
(1) In Table 1, our method HyperCore outperforms all the baselines on MIMIC-III dataset. For example, compared with the state-of-the-art model DR-CAML, our method achieves  $2.2\%$  and  $3\%$  improvements of Micro-F1 score on MIMIC-III full and MIMIC-III 50 respectively. It indicates that, as compared to neural network based models that handle each code in isolation, our method can better take advantage of the rich correlations among codes. In addition, the small standard deviations indicate that our model obtains stable good results.
(2) As previous work (Mullenbach et al., 2018), the Macro-F1 score of our method on MIMIC-III full is lower than that on the MIMIC-III 50. The reason is that MIMIC-III full has long-tail frequency distributions, and the Macro-F1 places more emphasis on rare code prediction. Therefore, it is difficult to achieve a high Macro-F1 score on MIMIC-III full. Nevertheless, our method still achieves the best result on the Macro-F1 metric. It indicates that our method is very effective.
(3) In Table 2, our method HyperCore also achieves the best performance over all metrics on the MIMIC-II. Especially, compared with the state-of-the-art model DR-CAML, our method achieves  $5.9\%$  improvements of Macro-AUC, which indicates the effectiveness of our method.
(4) As shown in Table 2, the neural network based methods outperform the traditional model (SVM), which indicates the limitation of human
designed features and the advancement of neural networks for the automatic ICD coding.
4.5 Ablation Experiment
To investigate the effectiveness of the hyperbolic and co-graph representation, we conduct the ablation studies. The experimental results are listed in Table 3. From the results, we can observe that:
(1) Effectiveness of Hyperbolic Representation. Compared with the model removed hyperbolic representation, the HyperCore improves the Micro-F1 score from 0.539 to 0.551 on MIMIC-III full dataset. It demonstrates the effectiveness of the hyperbolic representation.
(2) Effectiveness of Co-graph Representation. Compared with the model removed the co-graph representation, the HyperCore model improves the performance, achieving  $2.6\%$  improvements of Micro-F1 score on the MIMIC-III 50 dataset. The great improvements indicate the co-graph representation is very effective.
(3) Effectiveness of Hyperbolic and Co-graph Representation. When we remove the hyperbolic and co-graph representation, the performance drops significantly. The Micro-F1 score drops from 0.477 to 0.439 on the MIMIC-II dataset. It indicates that simultaneously exploiting the hyperbolic and co-graph representation is also very effective.
4.6.1 The Analysis of Hyperbolic Code Embedding Dimension
Since the dimensionality of the hyperbolic code embeddings is very important for hyperbolic representation, we investigate its effect. The size of hyperbolic code embeddings is set 10, 20, 50, 70 and 100. Table 4 shows the results of our model on the MIMIC-III and MIMIC-II datasets. We have two important observations:
(1) The best hyperbolic code embedding dimensionality on MIMIC-III full is larger than it on MIMIC-III 50 and MIMIC-II. The reason may be that the number of codes in MIMIC-III full is
Dimensionality	MIMIC-III full	MIMIC-III 50	MIMIC-II
Macro-F1	Micro-F1	P@8	Macro-F1	Micro-F1	P@5	Macro-F1	Micro-F1	P@8
10	0.083	0.539	0.701	0.593	0.651	0.619	0.064	0.463	0.528
20	0.085	0.542	0.704	0.598	0.656	0.625	0.066	0.471	0.532
50	0.087	0.547	0.708	0.609	0.663	0.632	0.070	0.477	0.537
70	0.090	0.551	0.722	0.605	0.660	0.627	0.065	0.473	0.534
100	0.083	0.548	0.710	0.602	0.659	0.625	0.064	0.473	0.530
Table 4: Experimental results of HyperCore with different size of hyperbolic code embeddings.
ICD-9 code	Norm
460-519 (Diseases of the Respiratory System)	0.455
480-488 (Pneumonia and Influenza)	0.520
487 (Influenza)	0.568
487.8 (Influenza with other manifestations)	0.928
520-579 (Diseases of the Digestive System)	0.412
550-579 (Hernia of Abdominal Cavity)	0.472
550 (Inguinal hernia)	0.590
550.0 (Inguinal hernia with gangrene)	0.902
more than other two datasets, which needs higher-dimensional hyperbolic code embedding to represent the code hierarchy.
(2) The performance does not always improve when the hyperbolic code embedding size increases. We guess that low dimensional embeddings can capture the hierarchy and the network is prone to over-fitting when high dimensional hyperbolic code embeddings are used.
4.6.2 The Hierarchy of Hyperbolic Code Embedding
After embedding the ICD codes into the hyperbolic space, the top level codes will be placed near the origin and low level codes near the boundary, which can be reflected via their norms. Table 5 shows examples of ICD-9 codes and their hyperbolic norms. The first and second blocks list codes of "Diseases of the Respiratory System" and "Diseases of the Digestive System", respectively. As expected, the lower level codes have higher hyperbolic norms, and this approves that when the disease is more specific, the hyperbolic norm is larger. For example, code "487.8 (influenza with other manifestations)" has a higher norm than "487 (influenza)", and "550.0 (inguinal hernia with gangrene)" has a higher norm than "550 (inguinal hernia)". It indicates that the hyperbolic code embeddings can
Table 5: The first and second blocks list some codes and their hyperbolic norms of "Diseases of the Respiratory System" and "Diseases of the Digestive System", respectively. In each block, the disease becomes more specific from top to bottom. The norms of codes increase with the depth.
Input	Mr. [**Known firstname 58216**] is an 87 year old male with a h/o Parkinsons Disease, difficulty breathing, ..., 87 year old male presents with severe chest tightness, respiratory failure, and pneumatosis coli indicative of visceral necrosis. As the patient was not a surgical candidate, medical prognosis was poor .......
Gold Label	518.81; 401.9; 276.2; 038.9
CNN+Attention	518.81; 401.9; 518.83; 518.84
HyperCore	518.81; 401.9; 276.2; 038.9
Figure 4: An example to illustrate the effectiveness of the proposed model. The green bold codes indicate they are highly correlated. The red bold codes denote there exists contradictions between them.
capture the code hierarchy.
We give an example shown in Figure 4 to illustrate the visualization of code-wise attention and the effectiveness of hyperbolic and co-graph representation. (1) Code-wise attention visualization: When the HyperCore model predicts the code "518.81 (acute respiratory failure)", it can assign larger weights to more informative words, like "respiratory failure" and "chest tightness". It shows the codes-wise attention enables to select the most informative words. (2) The effectiveness of hyperbolic representations: Our proposed model and the CNN+Attention can both correctly predict the code "518.81". However, the CNN+Attention model gives contradictory predictions. Our proposed model can avoid the prediction contradictions by exploiting code hierarchy, which proves the effectiveness of hyperbolic representations. (3) The effectiveness of co-graph representation: Although there is no very obvious clue to predict the code "276.2 (acidosis)", our model can exploit the co-occurrence between the code "518.81" and "276.2" to assist in inferring the code "276.2". It demonstrates the effectiveness of the co-graph representation.
In this paper, we propose a novel hyperbolic and cograph representation framework for the automatic ICD coding task, which can jointly exploit code hierarchy and code co-occurrence. We exploit the hyperbolic representation learning method to leverage the code hierarchy in the hyperbolic space. Moreover, we use the graph convolutional network to capture the co-occurrence correlation. Experimental results on two widely used datasets indicate that our proposed model outperforms previous state-of-the-art methods. We believe our method can also be applied to other tasks that need to exploit hierarchical label structure and label co-occurrence, such as fine-grained entity typing and hierarchical multi-label classification.
This work is supported by the Natural Key R&D Program of China (No.2017YFB1002101), the National Natural Science Foundation of China (No.61922085, No.61533018, No.61976211, No.61806201) and the Key Research Program of the Chinese Academy of Sciences (Grant NO. ZDBS-SSW-JSC006). This work is also supported by Beijing Academy of Artificial Intelligence (BAAI2019QN0301) and the CCF-Tencent Open Research Fund.
Anand Avati, Kenneth Jung, Stephanie Harman, Lance Downing, Andrew Ng, and Nigam H Shah. 2018. Improving palliative care with deep learning. BMC medical informatics and decision making, 18(4):122.
Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Simaan. 2017. Graph convolutional encoders for syntax-aware neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1957-1967.
Tal Baumel, Jumana Nassour-Kassis, Raphael Cohen, Michael Elhadad, and Noemie Elhadad. 2018. Multi-label classification of patient notes: case study on icd code assignment. In *Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence*.
Silvere Bonnabel. 2013. Stochastic gradient descent on riemannian manifolds. IEEE Transactions on Automatic Control, 58(9):2217-2229.
Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F Stewart, and Jimeng Sun. 2016.
Doctor ai: Predicting clinical events via recurrent neural networks. In Machine Learning for Healthcare Conference, pages 301-318.
Bhuwan Dhingra, Christopher Shallue, Mohammad Norouzi, Andrew Dai, and George Dahl. 2018. Embedding text in hyperbolic spaces. In Proceedings of the Twelfth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-12), pages 59-69.
Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro, et al. 2018. Hyperbolic attention networks. In Proceedings of International Conference on Learning Representations.
Matthias Hamann. 2018. On the tree-likeness of hyperbolic spaces. In *Mathematical Proceedings of the Cambridge Philosophical Society*, volume 164, pages 345-361. Cambridge University Press.
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimiciii, a freely accessible critical care database. Scientific data, 3:160035.
Vianney Jouhet, Georges Defossez, Anita Burgun, Pierre Le Beux, P Levillain, Pierre Ingrand, and Vincent Claveau. 2012. Automated classification of free-text pathology reports for registration of incident cases of cancer. Methods of information in medicine, 51(03):242-251.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. In Proceedings of International Conference on Learning Representations.
Bevan Koopman, Guido Zuccon, Anthony Nguyen, Anton Bergheim, and Narelle Grayson. 2015. Automatic icd-10 classification of cancers from free-text death certificates. International journal of medical informatics, 84(11):956-965.
Dmitri Krioukov, Fragkiskos Papadopoulos, Maksim Kitsak, Amin Vahdat, and Marián Boguná. 2010. Hyperbolic geometry of complex networks. *Physical Review E*, 82(3):036106.
Dee Lang. 2007. Consultant report-natural language processing in the health care industry. Cincinnati Children's Hospital Medical Center, Winter, 6.
Leah S Larkey and W Bruce Croft. 1996. Combining classifiers in text categorization. In SIGIR, pages 289-297. CiteSeer.
Federico López, Benjamin Heinzerling, and Michael Strube. 2019. Fine-grained entity typing in hyperbolic space. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 169–180. Association for Computational Linguistics.
Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1506-1515.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. 2018. Explainable prediction of medical codes from clinical text. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1101-1111.
Maximillian Nickel and Douwe Kiela. 2017. Poincaré embeddings for learning hierarchical representations. In Advances in neural information processing systems, pages 6338-6347.
Maximillian Nickel and Douwe Kiela. 2018. Learning continuous hierarchies in the lorentz model of hyperbolic geometry. In International Conference on Machine Learning, pages 3776-3785.
Adler Perotte, Rimma Pivovarov, Karthik Natarajan, Nicole Weiskopf, Frank Wood, and Noémie Elhadad. 2013. Diagnosis code assignment: models and evaluation metrics. Journal of the American Medical Informatics Association, 21(2):231-237.
Aaditya Prakash, Siyuan Zhao, Sadid A Hasan, Vivek Datla, Kathy Lee, Ashequl Qadir, Joey Liu, and Oladimeji Farri. 2017. Condensed memory networks for clinical diagnostic inferencing. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pages 3274-3280. AAAI Press.
Anthony Rios and Ramakanth Kavuluru. 2018. Few-shot and zero-shot multi-label learning for structured label spaces. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3132-3142.
Haoran Shi, Pengtao Xie, Zhiting Hu, Ming Zhang, and Eric P Xing. 2017. Towards automated icd coding using deep learning. arXiv preprint arXiv:1711.04075.
Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018. Hyperbolic representation learning for fast and efficient neural question answering. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 583-591. ACM.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. In Proceedings of International Conference on Learning Representations.
Pengtao Xie and Eric Xing. 2018. A neural architecture for automated icd coding. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1066-1076.
Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Parek, Krishnan Srinivasan, and Dragomir Radev. 2017. Graph-based neural multi-document summarization. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 452-462.
Ying Yu, Min Li, Liangliang Liu, Zhihui Fei, Fang-Xiang Wu, and Jianxin Wang. 2019. Automatic icd code assignment of chinese clinical notes based on multilayer attention birnn. Journal of biomedical informatics, 91:103114.