# Enhancing Machine Learning Models Through PCA, SMOTE-ENN, and Stochastic Weighted Averaging
## 4 Discussion  
This study presents a comprehensive ensemble-based approach to enhancing predictive performance in classification tasks, with a focus on the Titanic dataset. By integrating seven diverse machine learning models--Logistic Regression, Support Vector Machine (SVM), Random Forest, XGBoost, LightGBM, and CatBoost--into a unified ensemble, the model's robustness and accuracy were significantly improved. The combination of Stacking and Voting strategies leverages the strengths of each individual model, allowing the ensemble to capture complex data patterns more effectively than any single model could.  
Furthermore, the application of Stochastic Weight Averaging (SWA) was pivotal in improving the generalization of the ensemble model. SWA averaged the model weights over multiple training cycles, reducing the likelihood of overfitting and enabling the model to generalize better to unseen data. This step was crucial, as models without SWA showed a tendency to perform well on training data but were less robust when applied to new data. After applying SWA, the model exhibited a substantial improvement in test accuracy, precision, and other key performance metrics.  
The key findings of this study highlight the ensemble model's effectiveness, with notable improvements in accuracy, AUC, and Average Precision after the application of SWA. Specifically, the ensemble model's accuracy increased from 91.97% to 94.89%, while the AUC and Average Precision both reached 0.98. These improvements indicate that the combination of Stacking, Voting, and SWA provides a highly effective framework for building accurate and reliable classification models.  
To provide a comprehensive evaluation of our model's performance, we compared our results with those of several recent studies using the Titanic dataset. [PERSON] and [PERSON] (2023) [PERSON] and [PERSON] (2023) implemented an ensemble approach combining Random Forest, SVM, and Gradient Boosting, achieving an accuracy of 81.3%. In another study, [PERSON] (2024) [PERSON] (2024) compared AdaBoost, XGBoost, and Random Forest with Random Forest, yielding a highest accuracy of 83.205%. [PERSON] et al. (2023) [PERSON] et al. (2023) also performed a comparative analysis using various machine learning models, reporting an accuracy of 81.10% with a Gradient Boost Trees model. Finally, [PERSON] (2024) [PERSON] (2024) achieved an accuracy of 87.5% with XGBoost and Random Forest.  
In comparison, our model, which incorporates dimensionality reduction (PCA), imbalanced data handling (SMOTE-ENN), and ensemble learning enhanced by Stochastic Weighted Averaging (SWA), achieved an accuracy of 94.89% and an AUC of 0.98. These results highlight the effectiveness of our hybrid approach in handling data imbalance and optimizing prediction accuracy, outperforming other models in terms of both accuracy and generalization.  
This paper makes comparisons with works cited in references [PERSON] and [PERSON] (2023); [PERSON] et al. (2023); [PERSON] et al. (2023), which are studies that used machine learning methodologies on survival prediction tasks. However, while our case study is comparable to these works, the methodology we propose is more general and not limited to survival prediction alone. The same steps can be adapted for other classification tasks, particularly those involving class imbalance, missing data, or complex feature interactions.