# Enhancing Machine Learning Models Through PCA, SMOTE-ENN, and Stochastic Weighted Averaging
###### Abstract  
Predicting survival outcomes in critical accidents has been a focal point in machine learning research. This study addresses several limitations of existing methods, including insufficient management of data imbalance, lack of emphasis on hyperparameter tuning, and proneness to overfitting. Many existing models struggle to generalize effectively on imbalanced datasets or depend on default hyperparameter settings, resulting in biased predictions. By integrating Principal Component Analysis (PCA), hyperparameter optimization, and resampling methods, as well as combining Edited Nearest Neighbors (_ENN_) with the Synthetic Minority Oversampling Technique (SMOTE), the model significantly improves predictive accuracy and model generalization. An ensemble model combining seven machine learning algorithms--Logistic Regression, Support Vector Machine, KNN, Random Forest, XGBoost, LightGBM, and CatBoost--was applied to predict survival outcomes. Stochastic Weighted Averaging (SWA) was applied to mitigate overfitting and enhance generalization. The accuracy increased from 91.97% to 94.89% after SWA was applied in this specific scenario. The combination of PCA-based dimensionality reduction, hyperparameter tuning, and resampling techniques (_ENN_ + SMOTE) ensured the model handled data imbalance and optimized predictive accuracy. The final model demonstrated excellent performance, with Area Under the Curve (AUC) and Average Precision (AP) values both reaching 0.98, indicating high accuracy and precision. These improvements were validated using the Titanic dataset in a binary classification problem of predicting passenger survival. The results emphasize that ensemble learning, enhanced by SWA, offers a powerful framework for handling imbalanced and complex datasets, providing significant advancements in predictive modeling accuracy. This study provides insights into how machine learning techniques can be effectively combined to solve classification challenges in real-world scenarios. s +
Footnote †: journal: Medical  
This study applies Principal Component Analysis (PCA) alongside regularization techniques to mitigate overfitting and combines the Synthetic Minority Oversampling Technique (SMOTE) with the Edited Nearest Neighbors (_ENN_) technique (_ENN_ + SMOTE) to address data imbalance ([PERSON] et al., 2019). A comprehensive comparative analysis of three hyperparameter optimization techniques--grid search, random search, and Bayesian optimization--is performed, and an optimal ensemble approach combining Voting and Stacking methods, along with Stochastic Weighted Averaging (SWA) is utilized to achieve optimal performance ([PERSON] et al., 2019). Moreover, before applying the ensemble techniques, the weights were processed as a combined metric that maximizes the interval and accuracy of both Test Accuracy and Train Accuracy ([PERSON] et al., 2019). This approach ensures that the ensemble models are not only accurate but also well-generalized, balancing the trade-off between training and testing performance. By systematically comparing these methods, this study seeks to improve the predictive accuracy and generalizability of machine learning models in the context of survival prediction in serious accidents. This approach not only addresses the limitations observed in previous studies but also provides a more robust framework for predictive modeling in critical scenarios, offering significant contributions to the field of machine learning, particularly in optimizing predictive models for imbalanced and complex datasets ([PERSON] et al., 2019). This paper proposes a novel method for predicting the occurrence of serious accidents based on data-driven approaches. The contribution of this study is to find a hybrid prediction model with a machine learning approach that has high prediction accuracy, low computation time, and low memory usage.