## 6 Results and Discussion
### Performance of Our Models
The third prospective validation point was three months later (31 January 2021). During these three months, four additional patients had developed TLF for the first time. Our algorithm successfully found two of them, even though they were right-censored both in July and October and were used as TLF-negative samples for training the models. The other two underdiagnosed patients were in our not-at-risk group by the end of July. Similarly, we counted these two patients as FN during January validation. For the remaining 153 still at-risk patients, we updated their risk score to accommodate for the 90 days that had passed since the last calculation of the risk score in October. Passing through the same risk \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{
\begin{tabular}{c} **TNR** \\ **Specificity** \\ \end{tabular} } & \multirow{2}{*}{**FPR**} & \multicolumn{3}{c}{**TPR**} & \multirow{2}{*}{**Precision**} & \multirow{2}{*}{**F1**} & \multirow{2}{*}{**AUC-ROC**} \\  & & & & & **Sensitivity** & & \\ \hline ERT & 0.81 \(\pm\) 0.02 & 0.19 \(\pm\) 0.02 & 0.66 \(\pm\) 0.07 & 0.34 \(\pm\) 0.07 & 0.13 \(\pm\) 0.02 & 0.18 \(\pm\) 0.03 & 0.62 \(\pm\) 0.01 \\ \hline GMB & 0.9 \(\pm\) 0.02 & 0.1 \(\pm\) 0.02 & 0.82 \(\pm\) 0.06 & 0.18 \(\pm\) 0.06 & 0.12 \(\pm\) 0.02 & 0.14 \(\pm\) 0.03 & 0.62 \(\pm\) 0.01 \\ \hline GP & 0.77 \(\pm\) 0.03 & 0.23 \(\pm\) 0.03 & 0.6 \(\pm\) 0.07 & 0.4 \(\pm\) 0.07 & 0.12 \(\pm\) 0.01 & 0.19 \(\pm\) 0.02 & 0.63 \(\pm\) 0.01 \\ \hline KNN & 0.21 \(\pm\) 0.02 & 0.79 \(\pm\) 0.02 & 0.09 \(\pm\) 0.04 & 0.91 \(\pm\) 0.04 & 0.09 \(\pm\) 0.01 & 0.16 \(\pm\) 0.01 & 0.62 \(\pm\) 0.01 \\ \hline L1-LR & 0.86 \(\pm\) 0.03 & 0.14 \(\pm\) 0.03 & 0.76 \(\pm\) 0.03 & 0.24 \(\pm\) 0.03 & 0.13 \(\pm\) 0.02 & 0.17 \(\pm\) 0.02 & 0.62 \(\pm\) 0.01 \\ \hline L2-LR & 0.62 \(\pm\) 0.04 & 0.38 \(\pm\) 0.04 & 0.48 \(\pm\) 0.05 & 0.52 \(\pm\) 0.05 & 0.1 \(\pm\) 0.01 & 0.17 \(\pm\) 0.01 & 0.63 \(\pm\) 0.01 \\ \hline MLP & 0.82 \(\pm\) 0.08 & 0.18 \(\pm\) 0.08 & 0.69 \(\pm\) 0.1 & 0.31 \(\pm\) 0.1 & 0.13 \(\pm\) 0.02 & 0.18 \(\pm\) 0.02 & 0.62 \(\pm\) 0.01 \\ \hline RF & 0.94 \(\pm\) 0.02 & 0.06 \(\pm\) 0.02 & 0.89 \(\pm\) 0.03 & 0.11 \(\pm\) 0.03 & 0.13 \(\pm\) 0.04 & 0.11 \(\pm\) 0.03 & 0.63 \(\pm\) 0.01 \\ \hline SVM & 0.76 \(\pm\) 0.05 & 0.24 \(\pm\) 0.05 & 0.65 \(\pm\) 0.08 & 0.35 \(\pm\) 0.08 & 0.11 \(\pm\) 0.01 & 0.16 \(\pm\) 0.01 & 0.62 \(\pm\) 0.01 \\ \hline Majority voting * & 0.81 & 0.19 & 0.66 & 0.34 & 0.13 & 0.19 & 0.62 \(\pm\) 0.01 \\ \hline Mean probability * & 0.80 & 0.20 & 0.66 & 0.34 & 0.12 & 0.18 & 0.63 \(\pm\) 0.01 \\ \hline SL * (early-term) & 0.61 & 0.39 & 0.46 & 0.54 & 0.10 & 0.17 & 0.62 \(\pm\) 0.01 \\ \hline SL * (late-term) & 0.87 & 0.13 & 0.47 & 0.53 & 0.27 & 0.36 & NA \\ \hline SL * (very late-term) & 0.92 & 0.08 & 0.47 & 0.53 & 0.36 & 0.43 & NA \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Performance of the nine individual ML models in our study, three ensemble learners, and prospective validations for prediction of early-, late-, and very late-term TLF. * = calculated on sum of confusion matrices. NA = AUC-ROC is not measurable for these two-step follow-up models. score cut-off, at the end of January, our algorithm flagged 105 patients as still at risk and moved 48 to the not-at-risk group. By the time of the submission of this paper, no new patients were reported with TLF. This put our potential FPR at 8.44%. At this point, we achieved a performance of TNR = 91.66% and TPR = 52.76%.