## 4 Empirical Assessment Using Real-World Data
### Results  
#### 4.3.1 Classification Stage  
In the classification stage of our analysis, the data were strategically split into training and testing sets at an 80:20 ratio, employing stratified sampling to maintain the proportion of the minority class. This sampling strategy ensured that the model was tested against a realistic distribution of crash instances, which is essential when the positive class (crash occurrence) represents a smaller portion of the data. By doing so, we mitigated the risk of developing a biased model that overlooks the less frequent, yet critical, positive cases. The analysis was conducted within the Python 3.8 environment, employing the Scikit-learn library, a robust tool for machine learning analysis. Table 4 shows a summary of each model's classification performance. Logistic regression showed a high accuracy of 0.82 but had a low recall of 0.15, indicating that it was not as effective in identifying the minority class, which is critical in crash prediction. Decision trees demonstrated an accuracy of 0.74 with a recall of 0.32, suggesting a better performance than logistic regression in detecting the minority class, but at the cost of overall accuracy. Random forest improved upon decision trees' performance, with an accuracy of 0.81 and a recall of 0.25. XGBoost presented similar accuracy to logistic regression, at 0.82, but improved recall, at 0.28, minimally balancing overall accuracy with minority class detection. The CatBoost model with the default loss function showed a slight improvement over XGBoost, with a recall of 0.26 but a higher accuracy of 0.83. Notably, the CatBoost model with a custom loss function demonstrated a significant leap in performance, with a recall of 0.77 and precision of 0.81, while maintaining an accuracy of 0.81. These numbers indicate a robust ability to detect the minority class without sacrificing the accuracy of the model. The recall metric is particularly informative for imbalanced datasets like ours, as it measures the model's ability to identify the positive class (crash occurrences). A high recall is valuable in practical applications, where failing to predict a crash could have serious consequences. Despite a slight compromise on accuracy, the customized CatBoost model's exceptional recall suggests that it is highly adept at detecting true crash events, outweighing the trade-off of a marginally lower accuracy score. Additionally, the F1 score provides a comprehensive measure of the balance between precision and recall. \begin{table}
\begin{tabular}{l c c c c} \hline
**Classifier** & **Accuracy** & **Precision** & **Recall** & **F1 Score** \\ \hline Logistic Regression & 0.82 & 0.56 & 0.15 & 0.23 \\ Decision Tree & 0.74 & 0.31 & 0.32 & 0.31 \\ Random Forest & 0.81 & 0.5 & 0.25 & 0.33 \\ XGBoost & 0.82 & 0.51 & 0.28 & 0.36 \\ CatBoost (Default Loss Function) & 0.83 & 0.58 & 0.26 & 0.35 \\ CatBoost (Custom Loss Function) & 0.81 & 0.81 & 0.77 & 0.78 \\ \hline \end{tabular}
\end{table} 
Table 4: Confusion matrix. Despite its high accuracy, logistic regression had a low F1 score of 0.24, reflecting its poor recall. Decision trees had an F1 score of 0.31, slightly improving on logistic regression but still indicating imbalances in performance. Random forest showed an F1 score of 0.33, representing a more balanced performance compared to decision trees. XGBoost, with an F1 score of 0.36, further balanced precision and recall. The CatBoost model with the default loss function had an F1 score of 0.36, indicating a slightly better balance than XGBoost. The CatBoost model with a custom loss function stood out, with a high F1 score of 0.79, underscoring its superior ability to balance precision and recall while effectively identifying crash occurrences. The receiver operating characteristic (ROC) curve and the corresponding plot of training and validation losses provide a deeper insight into the classification performance and the learning process of our custom CatBoost model.