# Enhancing Machine Learning Models Through PCA, SMOTE-ENN, and Stochastic Weighted Averaging
## 3 Experimental Results and Discussions
### Overfitting  
#### 3.3.1 Model Evaluation and Overfitting Detection  
The evaluated models include Logistic Regression, SVM, KNN, Random Forest, XGBoost, LightGBM, and CatBoost. The main performance metrics calculated and displayed are the area under the Receiver Operating Characteristic (ROC) curve and the Average Precision (AP) of the Precision-Recall (P-R) curve. Among the models, CatBoost exhibited the highest AUC (0.9411) and Average Precision (0.9269), making it the most effective model in terms of distinguishing between survivors and non-survivors with high precision. Random Forest also showed strong performance, with an AUC of 0.9289, though its lower Average Precision (0.8877) indicates it may be less accurate in predicting positive outcomes compared with CatBoost. Logistic Regression (AUC 0.9257, AP 0.9144) and LightGBM (AUC 0.9205, AP 0.9019) delivered balanced and reliable results, while SVM and KNN had the lowest AUCs (0.9045 and 0.8900, respectively), suggesting that these models may not be as effective, particularly when considering precision in positive predictions (Figure 4). In addition to AUC and Precision-Recall metrics, the models' performance was further evaluated using additional metrics such as accuracy, precision, recall, and F1 score, as outlined in Table 2. CatBoost consistently delivers strong performance, achieving an accuracy of 0.87, a precision of 0.88, and an F1 score of 0.84, reinforcing its superiority in handling this classification task. LightGBM also stands out with an accuracy of 0.86, while SVM and Random Forest demonstrate slightly lower accuracy but remain competitive. \begin{table}
\begin{tabular}{p{42.7 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**\#** & **Feature Name** & **Description** \\ \hline
0 & Age & Age of the passenger. Younger passengers had a higher survival likelihood. \\ \hline
1 & Fare & The fare paid for the ticket. Higher fares often indicated better survival chances. \\ \hline
2 & Title & Social status derived from the passenger’s name (e.g., Mr., Mrs., Miss). \\ \hline
3 & FamilySize & Size of the passenger’s family onboard, including siblings, spouses, parents, and children. \\ \hline
4 & Fare\_Per\_Person & The fare divided by family size, reflecting relative cost per person. \\ \hline
5 & Pclass\_1 & Binary variable for first-class passengers, who generally had better survival chances. \\ \hline
6 & Pclass\_2 & Binary variable for second-class passengers. \\ \hline
7 & Pclass\_3 & Binary variable for third-class passengers, who generally had lower survival rates. \\ \hline
8 & Sex\_Male & Binary variable indicating male passengers. Gender played a key role in survival. \\ \hline
9 & Sex\_Female & Binary variable indicating female passengers. \\ \hline
10 & Embarked\_C & Binary variable for passengers who embarked from Cherbourg. \\ \hline
11 & Embarked\_S & Binary variable for passengers who embarked from Southampton. \\ \hline
12 & Embarked\_Q & Binary variable for passengers who embarked from Queenstown. \\ \hline
13 & IsAlone & Binary feature indicating whether the passenger was traveling alone. \\ \hline
14 & Deck & Deck level of the passenger’s cabin. Higher decks may have offered better survival chances. \\ \hline
15 & FamilySurvival & Reflects the survival likelihood of family members onboard. \\ \hline
16 & Age Class & Interaction between age and class, capturing the combined effect on survival. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of the 17 Features Used for Survival Prediction. Interestingly, KNNs lags slightly behind in terms of both accuracy and F1 score, indicating that it may not be as well suited for this dataset. The overfitting assessment, visualized in Figure 5, highlights significant overfitting in models such as Random Forest and XGBoost, where training accuracies (98.74% and 98.03%) were much higher than test accuracies (84.36% and 83.24%, respectively). This indicates that these models tend to memorize the training data and struggle to generalize to new data.