# Enhancing Machine Learning Models Through PCA, SMOTE-ENN, and Stochastic Weighted Averaging
## 3 Experimental Results and Discussions
### Hyperparameter Optimization and Model Visualization  
#### 3.5.1 Hyperparameter Optimization  
This study uses three methods for hyperparameter tuning, grid search, random search, and Bayesian optimization, across seven machine learning models: Logistic Regression, Support Vector Machine (SVM), K-Nearest Neighbors (KNNs), Random Forest, XGBoost, LightGBM, and CatBoost. Hyperparameter tuning was consistently applied to all seven models. Figures 9 and 10 specifically illustrate the tuning process for logistic regression. The same tuning approach was applied to the other models, SVM, KNN, Random Forest, XGBoost, LightGBM, and CatBoost, even though the figures focus on logistic regression.  
Figure 9 presents a comparison of the accuracy achieved by each optimization method. First, grid search exhaustively explored all possible combinations of predefined hyperparameters, yielding an accuracy of 0.8702 and a mean absolute error (MAE) of 0.0803, albeit at a significant computational cost. Second, random search efficiently sampled random combinations, resulting in the same accuracy and MAE as grid search while requiring fewer computational resources. Finally, Bayesian optimization employed a probabilistic model to focus on the most promising regions of the hyperparameter space, achieving the highest accuracy of 0.8757 and a lower MAE of 0.0730, thus proving to be the most effective method in terms of both performance and computational efficiency. The best hyperparameters identified by Bayesian optimization included a moderate regularization strength (C = 0.0101) and a maximum iteration count of 121. Therefore, Bayesian optimization was determined to be the most effective approach for optimizing model performance while balancing computational cost.  
Figure 10 further illustrates the differences between training and test accuracy for each method, with arrows highlighting potential overfitting. Grid search and random search both showed a difference of \(-0.0495\) between training and test accuracy, while Bayesian optimization exhibited a slightly larger gap of \(-0.0513\). Despite this small difference, Bayesian optimization still demonstrated the lowest overall risk of overfitting, indicating better generalization to unseen data compared with the other methods. Therefore, Bayesian optimization was identified as the most effective method, balancing model complexity, generalization, and computational efficiency. Based on one's code and output, the result from Bayesian optimization will be returned as the best estimator.  
In this study, hyperparameter tuning was applied consistently across seven machine learning models: Logistic Regression, Support Vector Machine (SVM), K-Nearest Neighbors (KNN), Random Forest, XGBoost, LightGBM, and CatBoost. Each model underwent the same optimization process using grid search, random search, and Bayesian optimization to ensure comparability in performance evaluation. The goal was to identify the optimal set of hyperparameters for each model while balancing accuracy, mean absolute error (MAE), and computational efficiency. This uniform approach ensures that the models are evaluated on an equal footing, allowing for a fair comparison of their predictive performance and generalization ability.