# Enhancing Machine Learning Models Through PCA, SMOTE-ENN, and Stochastic Weighted Averaging
## 2 Materials and Methods
### Methodology
#### 2.2.3 Resampling  
In addressing the challenge of imbalanced datasets, this study employs several resampling techniques aimed at improving the classification models' performance. Six different resampling techniques (SMOTE, SMOTEENN, SMOTETomek, ADASYN, SMOTE + ADASYN, and _ENN_ + SMOTE) were applied to evaluate their effectiveness in addressing class imbalance. After comparison, _ENN_ + SMOTE was selected as the optimal resampling technique for the proposed methodology due to its superior performance. The other techniques were tested but not included in the final methodology. Additionally, experiments confirmed that the order of resampling, hyperparameter optimization, and ensemble learning significantly impacts performance, with the current order yielding the best results. Specifically, the methods were applied after feature engineering and before model training to ensure that the dataset was balanced before feeding it into the models. These resampling techniques--SMOTE, SMOTEENN, SMOTETomek, ADASYN, SMOTE + ADASYN, and _ENN_ + SMOTE--help mitigate class imbalance by generating synthetic minority class samples or cleaning noisy majority class samples. These methods are not specific to the Titanic dataset but are generally applicable to any dataset suffering from imbalanced class distributions, ensuring a more generalized methodology. SMOTE (Synthetic Minority Oversampling Technique): SMOTE is frequently used to balance data by creating synthetic examples for minority class samples through interpolation. This approach helps balance the class distribution, enabling the classifier to learn from a more representative sample of the data [19]. SMOTEENN (SMOTE + Edited Nearest Neighbors): SMOTEENN combines the synthetic generation capability of SMOTE with the data cleaning functionality of the Edited Nearest Neighbors (_ENN_) technique. SMOTE generates synthetic instances, while _ENN_ removes instances from the majority class that are incorrectly classified by their nearest neighbors, thus enhancing the classifier's performance by reducing noise and the risk of overfitting [20]. SMOTETomek: SMOTETomek combines SMOTE with Tomek links, a technique that identifies and removes instances from different classes that are each other's nearest neighbors. This process not only balances the class distribution but also cleans the overlapping instances between classes, improving decision boundaries for the classifier. ADASYN (Adaptive Synthetic Sampling): ADASYN is an adaptive method that generates synthetic instances for minority class samples that are more difficult to classify based on the distribution of the data. This method focuses on improving the classifier's ability to generalize on harder-to-learn instances, resulting in better performance, especially for datasets with complex decision boundaries. SMOTE + ADASYN: This technique combines SMOTE and ADASYN, first using SMOTE to generate synthetic instances and then applying ADASYN to further focus on hard-to-learn minority class examples, leading to improved generalization and model accuracy. ENN + SMOTE: This method combines the strengths of _ENN_ and SMOTE, where _ENN_ removes noisy or misclassified instances from the majority class, and SMOTE generatessynthetic instances for the minority class. This combination reduces noise while improving the class balance. The process of \(ENN\) is formalized as  
\[ENN(D)=\{x_{i}\in D\mid y_{i}\
eq majority(kNN(x_{i}))\}\]  
where \(D\) is the dataset, \(x_{i}\) represents an instance, and \(y_{i}\) is its label. The \(kNN(x_{i})\) function finds the nearest neighbors of \(x_{i}\) and the misclassified majority class instances are removed, resulting in a cleaner dataset. Next, SMOTE generates synthetic instances as  
\[x_{new}=x_{i}+\lambda\times(x_{NN}-x_{i})\]  
where \(x_{new}\) is the synthetic data point, \(x_{i}\) is the minority class instance, \(x_{NN}\) is one of the nearest neighbors, and \(\lambda\) is a random number between 0 and 1.