# Enhancing Machine Learning Models Through PCA, SMOTE-ENN, and Stochastic Weighted Averaging
## 2 Materials and Methods
### Methodology
#### 2.2.1 Feature Engineering and Principal Component Analysis (PCA)  
The first part of this study involves conducting data statistics for independent variables and preprocessing. The validity test of each variable was performed using exploratory data analysis (EDA) to ensure that relevant and significant features were considered. The dataset was divided into survival groups by removing redundant information and outliers. Based on the distribution of important features, such as age, fare, and embarkation location, additional independent variables were generated. This feature engineering process resulted in the creation of 17 variables, which provided a more comprehensive representation of the passengers' profiles and their likelihood of survival. The 17 variables included standard features like Passenger Class (Pclass), Age, Gender (Sex), and Fare, as well as newly engineered variables derived from existing ones, such as the number of family members on board, the interaction between socioeconomic class and age, and imputed values for missing data points like age. Additionally, the port of embarkation (Embarked) and fare ranges were categorized to better capture the impact of ticket price and boarding location on survival rates. Principal Component Analysis (PCA) was then applied to reduce the dimensionality of these variables, streamlining the model and reducing computational requirements. While PCA reduces the dimensionality of the feature space, improving computational efficiency, it also aids in mitigating overfitting by retaining only the most significant components. The ensemble of seven classifiers was selected for its ability to capture different aspects of the data, with models such as SVM excelling in high-dimensional spaces and Random Forests providing robustness against overfitting. The use of an ensemble classifier, particularly when combined with Stochastic Weighted Averaging (SWA), further enhances generalization by averaging predictions from diverse models, leading to improved performance compared with a single, finely tuned model. By converting the 17 original variables into a smaller set of independent components, PCA ensured that the most significant variance in the dataset was retained while minimizing the risk of overfitting. Principal Component Analysis (PCA) was utilized in this study to improve model performance in the Titanic survival prediction model. While PCA reduces dimensionality, it may limit the interpretability of individual features, which could be important in some contexts. However, in survival prediction systems such as this, interpretability is often secondary to predictive accuracy and performance. In this context, interpretability is not a critical requirement, as the goal is to maximize the model's accuracy in predicting survival outcomes. PCA effectively removes irrelevant data, highlighting key patterns while reducing overfitting and improving the model's generalization capabilities [10]. PCA is also valuable for reducing computational complexity, especially when dealing with large datasets. Although the computational cost increases when using an ensemble of multiple machine learning classifiers, this cost is justified by the significant gains in accuracy and robustness that ensemble methods provide. The added complexity is offset by the improved generalization and predictive performance achieved through the combination of PCA and ensemble learning [11]. Principal Component Analysis (PCA): A dimensionality reduction technique that transforms a large set of features into a smaller set while retaining as much of the original information as possible. PCA identifies the directions (principal components) in which the data varies the most and projects the data onto these new axes. The first principal component captures the largest amount of variance in the data, followed by the second component, and so on.