# Leveraging Language Models for Inpatient Diagnosis Coding
## 2 Materials and Methods
### Language Model Finetuning  
We formulated our problem as a multi-label classification task, where each instance can be assigned multiple labels simultaneously. In our case, the labels represent different ICD-10 codes that we aim to predict for a given document. We evaluated three pre-trained language models:  
* **Multilingual BERT**[17] or mBERT is a variant of the BERT model pre-trained on a multilingual corpus. It captures contextual information from words and sentences using a transformer architecture. Through training with different languages, the model learns to understand and generate representations for multiple languages. Multilingual BERT has been used in various NLP applications due to its ability to process multiple languages with a single model. * **Multilingual E5**[27] is a version of the E5 model that supports multiple languages. It was trained using a simple contrastive training approach. The training data for this model were collected from various sources on the Internet, resulting in an extensive text dataset. Multilingual E5 is a versatile embedding model that can be easily used for various tasks where a representation of text data is required. * **MEDPSU-RoBERTa** is our institutional language model with the RoBERTa architecture that was pre-trained on a comprehensive dataset of over 80 million clinical documents from Songklanagarind Hospital, Thailand. We adhered to pre-training guidelines, following the standard approach detailed in the original RoBERTa paper [28] with the transformer library [29]. In this way, MEDPSU-RoBERTa has been tailored to captures the nuances of clinical language allowing it can understand and analyse medical texts with a high degree of accuracy and precision. For each pre-trained language model, we removed the layers corresponding to the language model head, which are originally responsible for text generation. We then extended the pre-trained language models by using the output of the last hidden states for the [CLS] classification token and passing it through a classification head consisting of two dense layers (see Figure 4). Given that LM\({}_{\text{last}}\) represents the output of the last hidden state for the [CLS] classification token as a 768-dimensional vector, the output of the classification head can be calculated as:  
\[\hat{y}=L_{2}(D(\tanh(L_{1}(\text{LM}_{\text{last}})))) \tag{1}\]  
where \(L_{1}\) is the first dense layer with 768 hidden units, tanh is a hyperbolic tangent activation function, \(D\) is a dropout layer with a probability of 0.1 and \(L_{2}\) is the final dense layer with 8677 outputs, representing the output of the model \(\hat{y}\) containing raw prediction values. The sigmoid function is then applied to these raw prediction values to obtain probabilities, \(\hat{p}\), in the range of 0 and 1. Focal loss was used during the fine-tuning process to address the issue of imbalanced data. It addresses the problem of negative samples dominating the training process by assigning higher weights to difficult or misclassified samples. This weighting scheme  
Figure 4: Model architecture: The model utilizes data from four main fields: principal diagnosis, co-morbidities, procedures and treatment details. The language models were fine-tuned with two dense layers, each with output sizes of 768 and 8677, respectively. Sigmoid activation was employed to obtain the multilabel probabilities. thus emphasises the learning of hard-to-classify samples and reduces the influence of easy samples, resulting in better handling of class imbalances during training. Given \(y\in\{0,1\}\) is the binary ground truth and \(\hat{p}\in[0,1]\) is the predicted probability of the true class. The individual focal loss is define as  
\[\text{Focal Loss}= \begin{cases}-\alpha(1-\hat{p})^{\gamma}\log(\hat{p})&\text{ if }y=1\\ -(1-\alpha)\hat{p}^{\gamma}\log(1-\hat{p})&\text{if }y=0\end{cases}