## 2 Materials and Methods
### Machine Learning Models  
This study aims to predict categorical variables using supervised machine learning classification algorithms. The methodologies investigated include multinomial logistic regression, support vector machines (SVM), decision trees, random forests, and gradient boosting methods: gradient boosting machines (GBM), XGBoost, and CatBoost. Multinomial logistic regression (MLR) and decision trees (DT) represent two distinct approaches to classification. Multinomial logistic regression extends binary logistic regression to multi-class problems by modeling the probability of each categorical outcome using a linear combination of input features. It calculates the log odds of each class relative to a reference category and transforms these into probabilities using the softmax function, enabling simultaneous prediction across multiple classes. This approach allows coefficients to be interpreted as the effect of predictors on the log odds of an outcome relative to the reference category, making it valuable for understanding relationships between features and outcomes. Decision trees predict outcomes by recursively partitioning the data space based on feature splits. These splits are chosen to maximize class separation using impurity measures such as Gini impurity or entropy. Gini impurity measures the likelihood of misclassification within a node, while entropy quantifies uncertainty or disorder in class distributions. Both metrics aim to maximize information gain at each split, ensuring nodes become increasingly \"pure\" as the tree grows. However, decision trees are prone to overfitting due to their hierarchical structure and lack of regularization mechanisms. A random forest algorithm can be used to address these decision tree issues. Furthermore, gradient boosting, XGBoost, and CatBoost are boosting methods that build an ensemble model by combining several weak decision trees sequentially and correcting errors made by previous models. Figure 1: Feature Importance Comparison Using F-value and Mutual Information. Random forests aggregate predictions from multiple decision trees trained on bootstrapped samples of data, reducing overfitting through ensemble averaging and feature subsampling. The inherent regularization provided by random forests avoids explicit penalty terms, making them robust for classification tasks. Gradient boosting methods (GBM, XGBoost, and CatBoost) iteratively build weak decision trees that correct errors made by previous models. Both CatBoost and XGBoost incorporate explicit regularization during training--CatBoost uses I2_leaf_reg for L2 regularization on leaf values, while XGBoost employs lambda for L2 penalties on weights--to control model complexity and prevent overfitting. Hyperparameter tuning was performed to enhance the performance of the machine learning models and mitigate challenges such as overfitting, underfitting, and variability in predictions. While default hyperparameter settings offer a baseline, they often fail to fully capture the complexity of real-world datasets or leverage the full potential of the algorithms. For example, gradient boosting methods like CatBoost and XGBoost rely on parameters such as learning rate and tree depth to balance model complexity and stability during training. Similarly, ensemble methods like random forests benefit from tuning parameters such as the number of trees (n_estimators) and feature subset size (max_features) to improve generalization and reduce overfitting. To identify optimal configurations, we systematically explored the hyperparameter space using RandomizedSearchCV, aiming to maximize accuracy while minimizing variance across cross-validation folds. This process was conducted using 10-fold cross-validation to ensure that the tuning results were both reliable and generalizable.