# Application of Machine Learning Techniques to Predict a Patient's No-Show in the Healthcare Sector
## 5 Results and Discussion  
This section presents the results obtained after performing the data pre-processing, adjusting the class imbalance using SMOTE oversampling, performing feature selection, and tuning algorithms hyperparameters per classifier. The results illustrate the performance obtained when testing the models mentioned in Section 4 of this paper via multiple, defined metrics. To report on and evaluate model performance, we applied the following performance metrics [44]:  
1. Accuracy: a simple ratio of total correct predictions over total wrong predictions. 2. Precision (Positive predictive value): the ratio of correctly predicted instances per class to all predictions made for that same class. 3. Recall (Sensitivity): the ratio of the correctly predicted instances per class to the total amount of actual instances labelled to that class. 4. F1-score: a harmonic mean of recall and precision. These results are acquired by drawing a confusion matrix for each classifier and then taking the average of the results. The confusion matrix enables visualization of the metrics, where the diagonal elements represent the number of points for which the predicted label is equal to the true label, while off-diagonal elements are those that are mislabelled by the classifier. After training all the three algorithms and fitting the Random Forest model with the test data, we obtained the confusion matrix shown in Figure 17.  
In the considered case study, we are interested in predicting the high number of patients who could not attend medical appointments by minimizing the incidence of false negatives. Thus, we selected the Random Forest classifier as the best classification algorithm able to achieve the objective of the analysis. In Table 2, we summarize the classification report of the Random Forest model, which is related to precision, recall, and F1-score statistical metrics. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Precision** & **Recall** & **F1-Score** & **Support** \\ \hline
0 & 0.91 & 0.93 & 0.92 & 1469 \\
1 & 0.93 & 0.91 & 0.92 & 1468 \\ accuracy & & & 0.92 & 2937 \\ macro avg & 0.92 & 0.92 & 0.92 & 2937 \\ weighted avg & 0.92 & 0.92 & 0.92 & 2937 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Random Forest classifier metrics. Note, 0 = not attend and 1 = attend. Figure 16: Schematic overview of k-fold cross-validation [43]. After the holdout process in the oversampled data described in Section 4.2, we trained and tested the algorithm with 5963 and 2937 instances, respectively. The Random Forest algorithm correctly classified 2703 out of the total amount of test instances. Thus, the classifier obtained:  
* The highest true positive rate of approximately 92%, correctly predicting 1363 out of 1469 of the patients who do not attend the scheduled appointment;
* The lowest false positive rate of approximately 0.07%. It only failed to detect 106 patients who had attended the appointment, obtaining the best recall score of 0.91. The recall was identified as the most relevant performance metric to ensure the minimum number of false negatives (patients who may potentially attend the appointment but are not classified as such by the system) to a lack of precision resulted in considerable numbers of false positives. The Area Under the Curve (AUC) of Receiver Characteristic Operator (ROC) curve is an often-used performance metric for classification problems. It is one way to assess the rate of observed positives predicted as positives (sensitivity) and the proportion of observed negatives predicted as negatives (specificity). As closest to 1 the AUC is, better the model is at distinguishing between patients who will attend and not attend the appointment. When developing a machine learning model, it is equally important not only have an accurate, but also an interpretable model. Often, apart from wanting to know what our model's prediction is, we also wonder which features are most important in determining the result.