## 3 Results
### Trained Model Performance
Figure 4: AUC scores of the three trained models. A model that makes random guesses (practically a model with no discriminative power), is represented by the diagonal dashed blue line that extends from the bottom left (0, 0) to the top right (1, 1). The ROC curve for any model that outperforms the random one will be above this diagonal line. **General Potential Risk Areas for Overfitting:**  
* **Feature Engineering:** The extraction of high-quality landmarks from the images serve as features for the model. If too-specific details are being captured from these features, the model could overfit to the specific data and may not be able to generalize to new data. * **Clean Dataset and High Performance:** The dataset used for this model was within a sterile environment without much noise or artifacts. Despite this being a good case for our model training, leading it to perform exceptionally, it may cause it to perform less efficiently in real-world scenarios. Gradient Boosting and Random Forest achieve high performance metrics for F1 score, recall, precision and accuracy. Their ensemble nature that combines multiple decision trees makes them generally robust to overfitting [40; 41]. However, it is crucial to control overfitting and ensure that metrics are not misleading. In order to control overfitting, we included the following methods:  
**Imputation of Missing Values:**  
By handling missing values with simple imputation, we ensure that in cases of rows with incomplete data, the model does not overfit by discarding them entirely. The model is learning not just by one full row but from the entire dataset. **Training-Testing Split:**  
* The code splits the dataset into 80% of the data for training and 20% of the data for testing. Since the models are being trained on the training set and then evaluated on the test set that contains data unseen by the model during training, the performance results give a clear indication for a good model generalization. * The performance metrics on the test are also very high (see Table 2), suggesting that the model learned meaningful patterns rather than memorizing data from training. If a significant performance drop had occurred on the test set, then it would be an indication of overfitting. In our case, the model does not exhibit such behaviors. **Cross-Validation:**  
* We performed 5-fold cross-validation during our model's evaluation process. This technique is effective in controlling overfitting since it ensures that the model training is being evaluated on multiple subsets of the dataset, exposing it to different distributions of data. This prevents the model from memorizing patterns in a single set of training data. * Also, the evaluation metrics (F1 score, recall, precision and accuracy) are averaged across the validation folds with their standard deviations being reported, indicating how much variance in performance happened across different folds, thus providing insight into the consistency of the model's predictions across various data splits. High variance indicates potential overfitting. In our case, the low values on standard deviations suggest stable performance. * The results for each model after cross-validation are very similar, indicating that the performance is good across all the different folds of the data, which is a sign of avoiding overfitting. Further, the use of hyperparameter fine-tuning could help avoid overfitting in the future on a more complex dataset with higher levels of noise and artifacts. However, in our case, the hyperparameter tuning led to low ROC AUC scores and prediction accuracy, and was therefore omitted, as discussed later in the Section 3.3. L1 and L2 regularization techniques [42] prevent overfitting by discouraging the model of relying on a single feature by adding penalties to its complexity. Gradient Boosting and Random Forest are inherently resistant to overfitting due to their high performance in the generalization of the data, making the use of this technique not necessary in this case.