# Using Enhanced Representations to Predict Medical Procedures from Clinician Notes
## 2 Background  
When applying ML-based strategies, automatic ICD coding is formulated as a multi-label classification problem. A wealth of work has explored this topic using different ML-based approaches, some using conventional ML models and others integrating DL elements into their solutions. [PERSON] et al. [15] developed two prediction models based on Support Vector Machine (SVM), a flat classifier where each ICD code is predicted independently of any other, and another that exploits the hierarchical nature of the ICD codes, concluding that the latter yielded better coding results. [PERSON] et al. [16] applied an SVM-based solution to the specific task of classifying types of cancer from death certificates, allowing them to reduce the label space to predict. [PERSON] et al. [17] also resorted to conventional ML techniques, evaluating the performance of various classification models based on SVM, Logistic Regression (LR), and Multinomial Naive Bayes (MNB), in the task of assigning ICD codes to extensive medical records. Other works have focused on the hierarchical structure of codes, such as in the work of [PERSON] and [PERSON] [26], where they proposed a two-level hierarchical classification, a partial classification to first identify potential codes, followed by a second estimator that resolves the detailed codes. With the appearance of DL, many strategies have been developed in recent years to improve the results of traditional ML approaches. [PERSON] et al., [PERSON] et al., [PERSON] et al., and [PERSON] et al. [3; 10; 11] proposed different DL-based approaches to automatic ICD coding and compared their results against conventional ML alternatives, collectively showing better results for the DL models. [PERSON] et al. [10] developed a Convolutional Neural Network (CNN) model introducing per-label attention mechanisms, which boosted explainability by selecting the most relevant segments of the clinical notes. [PERSON] et al. [11] used transfer learning to adapt a Long Short-Term Memory (LSTM)-based architecture to predict diagnosis and procedure ICD codes. [PERSON] et al. [18] adapted an attention-based CNN and applied it to medical records in Brazilian Portuguese, demonstrating good performance in non-English applications. [PERSON] et al. [3] experimented with models based on LSTM and Gated Recurrent Unit (GRU), to capture long-term dependencies within the notes, some of which proved too long to retain useful information. They also considered a code hierarchy by simplifying the prediction task from specific codes to whole categories, which showed a significant improvement in accuracy at the expense of the level of detail of the final prediction. More recently, other techniques based on transformer architectures have shown promising results using text representations that exploit context. [PERSON] et al. [19] used word embeddings from different Bidirectional Encoder Representations from Transformers (BERT)-based models in a GRU classifier, showing improved performance compared to other embedding techniques. On the other hand, [PERSON] et al. [20] used a BERT-based model for automatic ICD coding and noted the limitations of the model when used with long pieces of text. However, despite the benefits of using transformer-based architectures in Natural Language Processing (NLP), these do not currently hold the SOTA in ICD prediction. As noted in [20], CNN and Recurrent Neural Networkt (RNN) alternatives can process text of any length, while their BERT-based counterparts are inherently limited in the number of words they can evaluate. In [27], the authors arrived at a similar conclusion, stating that one of the main limitations of the transformer model in this area is its limited ability to handle longer text sequences compared to other alternatives. Still, there are positives to the transformer approach, such as its ability to handle long-range dependencies or its improved interpretability compared to traditional CNNs, which may prove helpful in some instances.