# Using Enhanced Representations to Predict Medical Procedures from Clinician Notes
## 4 Implementation Design and Evaluation
### Results and Discussion  
In this section, we review the results achieved by the proposed approach to ICD procedure prediction, comparing with other works where applicable. Table 1 summarizes this evaluation, comparing the global micro \(F1\) score in the procedure prediction task for different _top-\(n_{P}\)_ code subsets, models, and text features. Our models are denoted by \({}^{\prime\uparrow\prime}\), and the best overall results for each ICD code subset are denoted by \({}^{\prime\ast\prime}\). The considered input types are defined as \(T_{V}\) for text vectors and \(ICD_{\{C,F\}}\) for the main categories of the diagnosis codes (\(C\) subscript) or the fully qualified codes (\(F\) subscript). Combined inputs are represented with combined symbols. Analyzing the performance of the models, we observed that text representations (\(T_{V}\)) based on contextual embeddings from BERT provided slight improvements in performance over the TF-IDF alternatives. This is in itself an interesting outcome, as the TF-IDF-based representations used in the experiments were significantly higher-dimensional, with vectors of 10,000 features compared to the 768-dimensional vectors from BERT-based embeddings. Furthermore, the BERT models were limited to sequences of 512 tokens, far from the average word count per document of 1551 (Figure 3). These results suggest that there might be sequential or temporal relationships between words in the text annotations that the BERT-based embeddings were able to capture and encode into considerably smaller representations. Although a significant part of the original text was left out, the resulting vector did a better job of capturing the core semantic meaning of the whole medical annotation, using a fraction of the features. Our results also show that there is indeed value in using diagnosis codes as part of the input (\(ICD_{\{C,F\}}\)), since there was a consistent boost in performance in every model when combining diagnosis codes with text vectors. Additionally, this boost increased with the level of detail of the diagnosis codes, and more interestingly, in some scenarios, it was still marginally improved when expanding the fully qualified code. We saw improvements in the \(F1\) score of up to 19% in our setup when using the most detailed diagnoses (\(ICD_{CF}\)). Even when considering only the less detailed representations corresponding to the main categories (\(ICD_{\text{C}}\)), there was still a noticeable performance improvement compared to using only medical annotations as input. This result suggests that even with limited knowledge about a patient's diagnosis, the proposed method can potentially assist in predicting the ICD codes for the most relevant procedures. Compared with other works, we have to consider some caveats regarding the scope of the prediction task and how we compared our work with the SOTA. Most studies tackled the problem of ICD code prediction by considering either the diagnosis codes alone or diagnoses and procedures, but without making clear distinctions between both groups in the prediction task. This dramatically limits the studies we can reliably compare to, as predicting procedure codes is clinically and functionally different from predicting diagnoses. Each group of codes represents different clinical stages and contains a wildly different number of codes, which leads to inconsistent results when comparing the two prediction tasks, as [PERSON] et al. showed in their results [10]. Considering previous considerations, we only compared our results with studies from the SOTA that explicitly reported results for predicting ICD-9 procedures. Specifically, we considered the work of [PERSON] et al. [10], in which they proposed a model for ICD prediction, CAML, based on a CNN architecture with attention mechanisms, and [PERSON] et al. [11], where they used a fine-tuned version of an AWD-LSTM architecture.