# Using Enhanced Representations to Predict Medical Procedures from Clinician Notes
## 4 Implementation Design and Evaluation
### Results and Discussion
Taking into account the best-performing models from the considered studies, the proposed method showed competitive results, outperforming the LSTM-based model in the top-10 prediction subtask with an _F1_ score of 0.697 versus their 0.690, and also in the case of the top-50 subtask, showing an _F1_ score of 0.632 versus the reported 0.614 of CAML. The best performing models are based on ClinicalBERT features combined with \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**ICD** & **Model** & \begin{tabular}{c} **Text** \\ **Features** \\ \end{tabular} & \begin{tabular}{c} **Micro F1 per Type of Input** \\ **\(T_{\text{V}}\)** \\ \end{tabular} & \begin{tabular}{c} **\(ICD_{\text{C}}T_{\text{V}}\)** \\ \end{tabular} & \
\begin{tabular}{c} \(ICD_{\text{F}}T_{\text{V}}\)** \\ \end{tabular} &
\begin{tabular}{c} \(ICD_{\text{CF}}T_{\text{V}}\)** \\ \end{tabular} \\ \hline \multirow{6}{*}{Top-10} & AWD-LSTM [11] & TF-IDF & 0.690 & - & - & - \\  & TF-IDF & 0.549 & 0.665 & 0.670 & 0.675 \\  & LR \({}^{\ddagger}\) & DistilBERT & 0.561 & 0.647 & 0.667 & 0.670 \\  & ClinicalBERT & 0.582 & 0.658 & 0.679 & 0.680 \\  & TF-IDF & 0.622 & 0.662 & 0.693 & 0.692 \\  & MLP \({}^{\ddagger}\) & DistilBERT & 0.553 & 0.643 & 0.670 & 0.673 \\  & ClinicalBERT & 0.571 & 0.653 & 0.679 & 0.697 \({}^{*}\) \\ \hline \multirow{6}{*}{Top-50} & LR [10] & Word2 Vec & 0.533 & - & - & - \\  & CAML [10] & Word2 Vec & 0.614 & - & - & - \\  & AWD-LSTM [11] & TF-IDF & 0.480 & - & - & - \\  & TF-IDF & 0.519 & 0.551 & 0.560 & 0.564 \\  & LR \({}^{\ddagger}\) & DistilBERT & 0.562 & 0.588 & 0.584 & 0.593 \\  & ClinicalBERT & 0.592 & 0.614 & 0.620 & 0.623 \\  & TF-IDF & 0.602 & 0.611 & 0.613 & 0.618 \\  & MLP \({}^{\ddagger}\) & DistilBERT & 0.566 & 0.578 & 0.585 & 0.594 \\  & ClinicalBERT & 0.599 & 0.606 & 0.632 \({}^{*}\) & 0.622 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of global micro _F1_ in procedure prediction. The proposed models are denoted by \({}^{*\pi*}\). The best overall results for each ICD subset are denoted by \({}^{**\pi*}\). an MLP classifier, showing marginally better results than the alternatives using an LR classifier and significantly better results when comparing ClinicalBERT with the other text representation alternatives. The proposed method showed a significant boost in performance, even when using the least detailed diagnosis codes, as seen when combining the text vectors with only the diagnosis categories \(ICD_{C}T_{V}\), where the model based on LR and embeddings from ClinicalBERT achieved comparable results to the CNN implementation from [10] with a comparable \(F1\) score of 0.614. When comparing similar setups, the proposed method also came out ahead. The model based on TF-IDF features and a LR classifier achieved an \(F1\) score of 0.564 when combining text annotations with diagnosis codes. These results outperformed those of the comparable LR with Word2 Vec features from [10], which reported a score of 0.533. The experiments showed that diagnosis codes and medical annotations can help predict relevant procedure codes, achieving competitive results that outperformed comparable methods in the SOTA. Our method also managed to achieve these results while relying on BERT-based models as a fundamental part of the text representation strategy, which, as other studies have pointed out [20; 27], currently does not hold the SOTA in predicting ICD codes from medical annotations. However, resorting to solutions based on transformers could be particularly beneficial in healthcare, due to the potential insights into the decision-making process their attention mechanisms can deliver [48].