SlowGuess/NAACL_2021_OCR huggingface dataset:
In all scenarios, our proposed reranking framework achieves consistent improvement over the base predictors except for LAAT, where the macro F-score on MIMIC-3 top-50 dev set slightly decreased. The relative improvement in macro F-score for all-code settings are more significant, ranging from  $1\%$  to  $16\%$ . Considering that the all-code setting is much more challenging and macro F-score is difficult to improve due to the data imbalance issue, the achieved improvement demonstrates the great potential of the proposed framework for better practicality. The MADE and Mask-SA reranker are both effective for the purpose of reranking. Their gains are similar across different settings and datasets. We believe that this trend is reasonable given that their formulations are similar, i.e., they both calculate score as product of conditional probabilities. We also observe that in the settings using all ICD codes, Mask-SA reranker provides larger improvement to the macro F-score consistently.
Our proposed framework improves upon the best-performing methods on all settings. Note that the proposed framework is complementary to the base predictor. The results show that our reranking method can improve upon any predictor that is designed with the independent assumption, demonstrating the great flexibility and generalizability of our method.
Model	Avg. Rank
LAAT	24.58
+ MADE	19.73
+ Mask-SA	19.50
Table 3: Average rank of the best-performing label set among the top-50 candidates.
4.5 Effect of Candidate Numbers
The reranking results reported in Table 1 and Table 2 are generated with top-50 candidates. In order to investigate the effect of number of candidates to the final performance, we plot the performance with regard to different number of candidates in Figure 4. As shown in the figure, the reranked score increases consistently when the number of candidates is less than 10. No significant improvement is observed when the number of candidates is larger than 10.
We hypothesize that this phenomenon is due to our formulation of the final score. When calculating the final scores, we combine the original score from the base predictor and the score from the reranker. For the candidates that originally ranked after 10 by the base predictor, the original score may be too low; hence it is almost impossible to be selected after reranking.
4.6 Effectiveness of Reranking
The ultimate goal of our reranker is to bring the best-performing label set to the highest rank. In order to further examine the effectiveness of our rerankers, we calculate the average ranking of the best-performing label set, i.e. the set with the highest micro F-score with respect to the ground truth, before and after reranking. The results are shown in Table 3, implying that the proposed model can bring the best candidate from the 24-th place to the 19-th place for better practicality in terms of the systems with doctors' interactions. Our rerankers improve the average ranking by more than  $20\%$  relative, demonstrating that the reranking process is effective.
4.7 Effect on Infrequent Labels
The task of ICD code prediction is extremely hard due to the large set of labels and the imbalanced data: the top-50 most frequent codes take up more than a third of all the outputs. To investigate the effect of the proposed framework on the infrequent labels, we bucket the labels according to their fre
Baseline + Rescoring
Sample 1(+ Mask-SA)	427.1 427.41 427.5 693.0 99.6 995.0427.1 427.41 427.5 693.0 99.6 995.0 99.62 96.04 96.71
Sample 2(+ MADE)	571.5 733.00 733.09 96.04 96.72 V66.7571.5 733.00 733.09 96.04 96.72 V66.7 305.1 431 96.6
Table 4: Sample results before and after reranking from MIMIC-3 data.
Figure 5: F1 scores  $(\%)$  with regard to the frequencies of labels. We bucket labels by their frequencies into 6 buckets.
quencies, and calculate the performance for each bucket. We plot the performance of MultiResCNN on MIMIC-3 full set with regard to label frequencies in Figure 5. The figure demonstrates that with reranking, the performance of the infrequent labels also increases. This result indicates that the reranking method is helpful for the extreme multi-label classification problem.
After comparing the results produced from two rerankers, we find that both methods have similar tendency of prediction. In other words, the original candidate sets would be improved by adding or deleting similar ICD codes after rescoring from both MADE and Mask-SA. To further analyze prediction change, Table 4 shows the original and reranked results for two data samples.
5.1 Addition and Deletion of Predictions
For the first sample, we find that the reranking module tends to add missing ICD codes to the predicted set. Specifically, the first sample has no 96 category in the original prediction, and the rescoring process adds 96.04 and 96.71 (highlighted in blue) in the candidate set for better performance. By checking their meanings, we could know that
96.04 is about insertion of endotracheal tube and 96.71 is about invasive mechanical ventilation, and both treatments are important for patients in ICU maintaining their respiratory function. Due to their strong dependency, we find that these codes frequently co-occur in the training data. Apparently, the reranker learn the correlation and is capable of improving the prediction in terms of both diversity and accuracy.
In the second sample, it can be found that our module can also help remove the unreasonable codes. Specifically, the code 733.09 (highlighted in red) is not proper to be the selected code due to the appearance of 733.00, which is the correct disease from the record. Therefore, the reranker can help not only provide additional accurate codes but also delete unreasonable ones for better performance.
5.2 Reranking Analysis
We further analyze our methods from the top-10 ranking candidates sets in the second sample to confirm if the sets with more accurate ICD codes would be at the top of the reranked sets. In this sample, the unreasonable ICD code 733.09 appears in every top-10 predictions before reranking. With reranking, our reranker is able to bring the set without 733.09 to the top-1. This example demonstrates that the reranker's ability to identify conflicting predictions and that we are able to correct them with the proposed framework.
This paper proposes a novel framework to improve multi-label classification for automatic ICD coding, which includes candidate generation and candidate reranking modules. In the first stage, a base predictor is performed to generate top-k probable label set candidates. In the second stage, we propose a reranker to capture the correlation between ICD labels without any external knowledge. Two types of the reranker, MADE and Mask-SA, are
employed to rerank the candidate sets. Our experiments show that both rerankers can consistently improve the performance of all predictors in MIMIC-2 and MIMIC-3 datasets, demonstrating the generalizability of our framework and the great potential of the flexible usage.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Tal Baumel, Jumana Nassour-Kassis, Raphael Cohen, Michael Elhadad, and Noemie Elhadad. 2018. Multi-label classification of patient notes: case study on icd code assignment. In _Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence_.
Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao, Shengping Liu, and Weifeng Chong. 2020. HyperCore: Hyperbolic and co-graph representation for automatic ICD coding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3105-3114, Online. Association for Computational Linguistics.
Weiwei Cheng, Eyke Hüllermeier, and Krzysztof J Dembczynski. 2010. Bayes optimal multilabel classification via probabilistic classifier chains. In ICML, pages 279-286.
Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F Stewart, and Jimeng Sun. 2016. Doctor ai: Predicting clinical events via recurrent neural networks. In Machine Learning for Healthcare Conference, pages 301-318.
Michael Collins and Terry Koo. 2005. Discriminative reranking for natural language parsing. Computational Linguistics, 31(1):25-70.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In ACL, pages 2978-2988, Florence, Italy. Association for Computational Linguistics.
Luciano RS de Lima, Alberto HF Laender, and Berthier A Ribeiro-Neto. 1998. A hierarchical approach to the automatic categorization of medical documents. In CIKM, pages 132-139. ACM.
Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. 2014. Large-scale object classification using label relation graphs. In ECCV, pages 48-64. Springer.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _NAACL_, pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. 2015. Made: Masked autoencoder for distribution estimation. In ICML, pages 881-889.
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimic-iii, a freely accessible critical care database. Scientific data, 3:160035.
Ramakanth Kavuluru, Anthony Rios, and Yuan Lu. 2015. An empirical evaluation of supervised learning approaches in assigning diagnosis codes to electronic medical records. Artificial intelligence in medicine, 65(2):155-166.
Cheng Li, Virgil Pavlu, Javed Aslam, Bingyu Wang, and Kochen Qin. 2019. Learning to calibrate and rerank multi-label predictions. In ECML-PKDD.
Cheng Li, Bingyu Wang, Virgil Pavlu, and Javed Aslam. 2016. Conditional bernoulli mixtures for multi-label classification. In ICML, pages 2482-2491.
Fei Li and Hong Yu. 2020. Icd coding from clinical text using multi-filter residual convolutional neural network. In AAAI.
James Mullenbach, Sarah Wiegrefe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. 2018. Explainable prediction of medical codes from clinical text. In NAACL, pages 1101-1111, New Orleans, Louisiana. Association for Computational Linguistics.
Kimberly J O'malley, Karon F Cook, Matt D Price, Kimberly Raiford Wildes, John F Hurdle, and Carol M Ashton. 2005. Measuring diagnoses: Icd code accuracy. Health services research, 40(5p2):1620-1639.
M. Ostendorf, A. Kannan, S. Austin, O. Kimball, R. Schwartz, and J.R. Rohlicek. 1991. Integration of diverse recognition methodologies through reevaluation of N-best sentence hypotheses. In Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove, California, February 19-22, 1991.
Adler Perotte, Rimma Pivovarov, Karthik Natarajan, Nicole Weiskopf, Frank Wood, and Noémie El-hadad. 2013. Diagnosis code assignment: models and evaluation metrics. Journal of the American Medical Informatics Association, 21(2):231-237.
Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. 2009. Classifier chains for multi-label classification. In ECML-KDD, pages 254-269. Springer.
Mohammed Saeed, Mauricio Villarroel, Andrew T Reisner, Gari Clifford, Li-Wei Lehman, George Moody, Thomas Heldt, Tin H Kyaw, Benjamin Moody, and Roger G Mark. 2011. Multiparameter intelligent monitoring in intensive care ii (mimic-ii): a public-access intensive care unit database. Critical care medicine, 39(5):952.
Federico Sangati, Willem Zuidema, and Rens Bod. 2009. A generative re-ranking model for dependency parsing. In Proceedings of the 11th International Conference on Parsing Technologies (IWPT'09), pages 238-241, Paris, France. Association for Computational Linguistics.
Libin Shen, Anoop Sarkar, and Franz Josef Och. 2004. Discriminative reranking for machine translation. In HLT-NAACL, pages 177-184.
Haoran Shi, Pengtao Xie, Zhiting Hu, Ming Zhang, and Eric P Xing. 2017. Towards automated icd coding using deep learning. arXiv preprint arXiv:1711.04075.
Shang-Chi Tsai, Ting-Yun Chang, and Yun-Nung Chen. 2019. Leveraging hierarchical category knowledge for data-imbalanced multi-label diagnostic text understanding. In LOUHI, pages 39-43, Hong Kong. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 5998-6008.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In ICML, pages 1096-1103.
Thanh Vu, Dat Quoc Nguyen, and Anthony Nguyen. 2020. A label attention model for icd coding from clinical text. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3335-3341. International Joint Conferences on Artificial Intelligence Organization.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Pengtao Xie and Eric Xing. 2018. A neural architecture for automated ICD coding. In ACL, pages 1066-1076, Melbourne, Australia. Association for Computational Linguistics.
Yinyuan Zhang, Ricardo Henao, Zhe Gan, Yitong Li, and Lawrence Carin. 2018. Multi-label learning from medical plain text with convolutional residual models. In Machine Learning for Healthcare Conference, pages 280-294.
Chenxi Zhu, Xipeng Qiu, Xinchi Chen, and Xuanjing Huang. 2015. A re-ranking model for dependency parser with recursive convolutional neural network. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1159-1168, Beijing, China. Association for Computational Linguistics.