# Leveraging Language Models for Inpatient Diagnosis Coding
## 3 Results  
All models were trained and validated using the training and validation data spanning the years 2016 to 2021. All performance measures presented in the table were calculated based on the predictions made by the models using the test data collected in the year 2022. This approach ensures an unbiased evaluation of the models' performance on unseen data and provides a realistic assessment of their generalisation capabilities. Table 1 shows a comparison of the performance metrics of the different models. The model finetuned from MEDPSU-RoBERTa, our institutional pre-trained language model, achieved the highest precision, highest recall, highest F1 score, highest exact match ratio, highest mean average precision, highest recall@5 and highest precision@5 compared to the other models. However, there is a slight difference in performance between the MEDPSU-RoBERTa, Multilingual BERT and Multilingual E5 models on various metrics. The running time for each model is impressively low. The basic model TF-IDF takes 17.29 ms to process one document, and the more advanced models such as Multilingual BERT, Multilingual E5 and MEDPSU-RoBERTa achieve even faster processing times of 4.66 ms, 4.76 ms and 4.81 ms, respectively. Looking at precision, recall and F1 score, the three models show relatively similar performance. The values for precision range from 0.8423 to 0.8439, for recall from 0.7149 to 0.7299 and for F1 score from 0.7735 to 0.7821. These results show that all three models have comparable abilities in detecting positive cases and minimising false positives and false negatives. For metrics such as exact match ratio, mean average precision, recall@5 and precision@5, the differences among the models are relatively small. The Friedman test revealed significant differences among the models (\(p<0.05\)) for recall, exact match ratio, F1 score, and mean AP (see Figure 5). Specifically, for recall, TF-IDF differs significantly from Multilingual E5 and MEDPSU-RoBERTa (\(p<0.05\)), while Multilingual BERT differs significantly from MEDPSU-RoBERTa (\(p<0.05\)). For the exact match ratio, TF-IDF differs significantly from Multilingual BERT and MEDPSU-RoBERTa (\(p<0.05\)), and Multilingual E5 differs significantly from MEDPSU-RoBERTa. In terms of the F1 score, all methods differ significantly from each other (\(p<0.05\)), except for Multilingual BERT and Multilingual E5, which do not exhibit a significant difference. Regarding mean AP, TF-IDF differs significantly from Multilingual E5 and MEDPSU-RoBERTa (\(p<0.05\)), while Multilingual BERT differs significantly from MEDPSU-RoBERTa (\(p<0.05\)). \begin{table}
\begin{tabular}{l l l l l l l l l} \hline \hline
**Model/Architecture** & **Run Time** & **Precision** & **Recall** & **F1 Score** & **Exact Match** & **Mean AP** & **Recall@5** & **Precision@5** \\ \hline
**TF-IDF (baseline)** & 17.29 ms \({}^{1}\) & **0.8456** & 0.5400 & 0.6590 & 0.2062 & 0.6928 & 0.5545 & 0.7085 \\
**Multilingual BERT** & 4.66 ms \({}^{2}\) & 0.8439 & 0.7149 & 0.7735 & 0.3411 & 0.7975 & 0.6126 & 0.7826 \\
**Multilingual E5** & 4.76 ms \({}^{2}\) & 0.8426 & 0.7155 & 0.7738 & 0.3380 & 0.8071 & 0.6155 & 0.7863 \\
**MEDPSU-RoBERTa** & 4.81 ms \({}^{2}\) & 0.8423 & **0.7299** & **0.7821** & **0.3470** & **0.8097** & **0.6181** & **0.7885** \\ \hline \hline \end{tabular}  
* Exact Match: Exact Match Ratio; Mean AP: Mean Average Precision; Bold values indicate the highest score for each metric; Run time measured under parallel computation in milliseconds (ms); \({}^{1}\) Inference performed in CPU; \({}^{2}\) Inference performed in GPU.  
\end{table} 
Table 1: Performance scores of all models calculated from the test set. Figure 6 shows a precision-recall curve illustrating the trade-off between precision and recall for all models. It shows how the performance of the model performance varies as the classification threshold changes. In our multi-label setting, we calculated the micro-averaged precision-recall curve from all labels for each model.