## 6 Results and Discussion
### Performance of Our Models  
Table 4 shows the performance of different components of our model which are validated both retrospectively and prospectively. The rows indicate the nine individual ML models, the three ensemble learners, and the prospective validation for prediction of early-, late-, and very late-term TLF. The nine individual models are: extremely randomized trees (ERTs), gradient boosting (GB), Gaussian process (GP), k-nearest neighbor (KNN), logistic regression with L1- and L2-regularization (L1-LR, L2-LR), multi-layer perceptron (MLP), random forest (RF), and support vector machines (SVMs). The three ensemble classifiers are: majority voting, mean probability, and super-learner. The results show that due to the nature of imbalanced class distributions, there is a steep trade-off between higher TP and TN. For example, KNN has the highest TPR (91%) but with very low TNR (21%). This means that KNN correctly predicts 91% of the TLF patients, but flags 79% of the healthy patients as at risk (FP = 1433), requiring clinicians to pay attention to a large cohort of patients. On the other hand, the RF classifier finds almost all (TNR = 0.94%) healthy patients correctly (TN = 1707) and generates the smallest at-risk cohort (FP = 117) to reduce clinicians' workload. Consequently, it misclassifies the majority of TLF patients (FNR = 89%). Looking at both ROC and PR curves (Figure 4), between the nine basic classifiers, GP performs best with the largest area under the curve (AUC). To statistically evaluate the performance of these classifiers, we applied an F-test. The null hypothesis was defined as 'there are no differences between the classification performances'. We first computed the F-statistic, then obtained the \(p\)-value from a cumulative F-distribution function. We rejected the null hypothesis with a \(p\)-value of <0.001 and an F-statistic of 585. The results showed we could not observe significant differences between the SL model and the other nine models for TLF prediction. In practice however, clinicians are interested in models with higher TPR while sacrificing moderate resources for FP patients. In this case, L2-LR with TPR = 54% and TNR = 62% is a better choice, generating 698 at-risk patients (FP = 698). We trained the SL only once at the end of Phase I to predict TLF at discharge. To analyze the performance of our method over time, we performed the first prospective TLFprediction analysis at the end of Phase I (31 July 2020). At this time point, our algorithm had flagged 705 patients as at risk, projecting a huge workload for the clinicians. To reduce the number of at-risk patients at this time point, we used the risk score model to generate a risk score for each patient, estimating their chance of TLF at that time point. We also chose an empirical cut-off value to reduce the number of at-risk patients to one-third (0.33).Therefore, we kept 234 patients in the at-risk group and moved the remaining 471 patients to the not-at-risk group. At this point, we achieved a performance with TNR = 61.35% and TPR = 54.30% (Figure 5). The second prospective validation point was three months later (31 October 2020). During these three months, eight patients had developed TLF for the first time (right-censored/healthy during the July cohort). Our algorithm successfully found two of them, even though they were right-censored in July and were used as TLF-negative samples for training the models. The other six underdiagnosed patients were correctly classified as not at risk for the end of July. We, however, counted these six patients as FN during the October validation. For the remaining 232 still at-risk patients, we updated their risk score to accommodate for the 90 days that had passed since the last calculation of the risk score in July. Passing through the risk score cut-off, at the end of October, our algorithm flagged 155 patients as still at risk and moved 77 to the not-at-risk group. At this point, we achieved a performance of TNR = 87.22% and TPR = 52.83%.