## 4 Discussion
### Hyperparameter Tuning Time & Performance  
For executing ML models and conducting statistical tests, hyperparameter tuning was performed to optimize both the 5 \(\times\) 2 cross-validation procedure (used for the 5 \(\times\) 2 CV paired \(t\)-test) and the 10-fold cross-validation approach (applied to all other statistical tests, except for the resampled test).  
To obtain 100 estimates, repeated 10-fold cross-validation with varying splits was conducted, ensuring alignment with the optimized hyperparameters for 10-fold CV. The only deviation from this optimized setup occurred in the corrected random resampled cross-validation paired \(t\)-test, where data splits followed a 66:34 ratio instead of the standard 90:10 used in other 10-fold cross-validation procedures. As a result, hyperparameters may not be fully optimized for this specific test.  
To analyze tuning time in relation to hyperparameter optimization results (Figure 2), a significant variation in tuning duration (time per trial) across different ML models can be observed. Notably, Random Forest and CatBoost require less tuning time under the 10-fold cross-validation procedure compared to the 5 \(\times\) 2 cross-validation approach, whereas ANN and XGBoost exhibit the opposite trend, requiring more time for 10-fold cross-validation, regardless of the fact that both methods involve the same total number of training and testing iterations (10 in total).  
Figure 6: Comparison of model execution times: mean fit and score times with standard deviation across machine learning models (100 estimates).  
However, the computational cost is not solely determined by the number of cycles; it also depends on the size of the training sets per each cycle. The size of the training and test sets differs between the two methods, which can impact computational time. In 10-fold cross-validation, each training set comprises 90% of the data, and each test set comprises 10%. In 5 \(\times\) 2 cross-validation, each training set comprises 50% of the data, and each test set also comprises 50%. Training on larger datasets generally requires more computational time.  
Therefore, the larger training sets in 10-fold cross-validation should lead to longer training times per iteration compared to 5 \(\times\) 2 cross-validation. However, the specific machine learning algorithm used can influence the computational time. Some algorithms scale differently with the size of the training data, which can affect the overall time required for cross-validation. This trade-off between smaller but repeated folds (5 \(\times\) 2 CV) and larger single-fold partitions (10\(\times\) CV) highlights the importance of aligning cross-validation selection with model complexity and resource availability.  
Furthermore, in terms of hyperparameter tuning performance for F1 (Figure 3), it was observed that 10-fold cross-validation generally yielded slightly better results. This improvement is likely attributed to the larger proportion of training data per fold, which facilitated more effective generalization. Random Forest exhibited a relatively smaller performance difference, but it required significantly more time under 5 \(\times\) 2 cross-validation due to the repeated use of data and its tendency to favor deeper ensemble structures. The smallest gap in performance between 5 \(\times\) 2 cross-validation and 10-fold cross-validation was observed in the Logistic Regression model, likely because it is less sensitive to variations in training data size due to its simplicity and lower risk of overfitting compared to more complex models.