# Predictive Modeling of Flight Delays at an Airport Using Machine Learning Methods
## 6 Results
### Performance Evaluation  
The effectiveness of the learning models was assessed during both the training and the testing phases for all trials. The success of the techniques was determined through a comprehensive evaluation using multiple performance metrics. These metrics offer various insights into a model's performance:  
Accuracy: This metric measures the overall correctness of a model's predictions and is widely used in the literature. It is computed from the confusion matrix. However, for imbalanced datasets, relying solely on accuracy can be misleading [66]. Recall (sensitivity): Recall represents the percentage of relevant instances that were correctly identified by the model. It is a vital metric for cases where finding all relevant instances is crucial. Precision (positive predictive value): Precision indicates the percentage of retrieved instances that are relevant to the problem at hand. High precision is important when minimizing false positives is essential. F-1 score: The F-1 score is the harmonic mean of precision and recall. It provides a balanced evaluation that considers both false positives and false negatives. Balanced error rate: This metric offers an assessment of error that considers class imbalances and can be particularly valuable for imbalanced datasets. These metrics collectively offer a robust evaluation of a techniques' performance, considering different aspects of predictive accuracy and suitability for the task. The result of the first trial, which was conducted with only the airport data, is shared in Table 5. XGBoost and CatBoost consistently demonstrated strong performance across multiple metrics in both the training and the testing phases, which makes them promising candidates for predicting flight delays. Logistic Regression, ANN, and LightGBM also delivered reasonable results, while Naive Bayes performed less effectively in most metrics. \begin{table}
\begin{tabular}{c c} \hline \hline
**Method** & **Parameters** \\ \hline Logistic Regression & penalty: (2, 1) \\  & C: (0.001, 0.01, 0.1, 1, 10, 100, 1000) \\ \hline Naive Bayes & var\_smoothing:(0, 0.9, num = 100) \\ \hline XGBoost & learning\_rate: (0.01, 1.0), n\_estimators: (100, 1000) \\  & max\_depth: (3, 15), subsample: (0, 1.0) \\  & gamma: (0, 5), min\_child\_weight: (0, 20) \\ \hline CatBoost & max\_depth: (5, 15), bagging\_temperature: (3, 10) \\  & l2\_leaf\_reg: (2, 10) \\ \hline Random Forest & n\_estimators: (100, 1000), max\_depth: (4, 40) \\  & min\_samples\_split: (2, 100), min\_samples\_leaf: (1, 20) \\ \hline LightGBM & num\_leaves: (20, 50), max\_depth: (5, 30) \\  & lambda\_l2: (0.0, 0.05), lambda\_l1: (0.0, 0.05) \\  & min\_child\_samples: (5, 100), min\_data\_in\_leaf: (5, 100) \\  & feature\_fraction: (0.1, 0.9), bagging\_fraction: (0.8, 1) \\ \hline ANN & hidden\_layer\_sizes: ((5,5), (5,10), (5,15), (5,20), (5,25), (10,5), (10,10), \\  & (10,15), (10,20), (10,25), (15,5), (15,10), (15,15), (15,20), (15,25), (20,5), \\  & (20,10), (20,15), (20,20), (20,25), (25,10), (25,15), (25,20), (25,25)) \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Experimental design for hyperparameter space. In the second experiment, the same dataset was balanced with SMOTE. The results are presented in Table 6. Overall, the application of the SMOTE technique helped improve recall in several models during the training phase, at the expense of precision in some cases. However, it did not consistently improve the model performance in the testing phase, with recall often decreasing. XGBoost, CatBoost, and LightGBM continued to demonstrate strong performance even with SMOTE, suggesting their robustness in dealing with both imbalanced and balanced data.