## 3 Methodology
### Feature Selection
#### 3.2.1 Elastic Net-Based Feature Selection  
Elastic Net is a regularization technique that combines the benefits of both Lasso and Ridge regression [30]. It leverages the L1 and L2 penalties, respectively, making it particularly well-suited for situations with multiple correlated features and high dimensionality. In general, Lasso Regression is effective for high-dimensional datasets with many potentially irrelevant features, while Ridge regression is particularly effective when dealing with highly correlated features. Elastic Net addresses the limitations of both methods by incorporating a mixing parameter (\(\alpha\)) that balances the two penalties. This allows it to handle datasets with multicollinearity more effectively by selecting groups of correlated features rather than a single representative feature, which is a common issue with Lasso regression [31]. When datasets exhibit both characteristics, employing Elastic Net can significantly enhance model performance.  
Elastic Net is an embedded-based feature selection method. The predicted values generated by Elastic Net are based on the linear combination of selected features and their corresponding coefficients, which are estimated during the model-training phase. These coefficients are optimized using a loss function that incorporates both L1 and L2 penalties. In cases where the number of predictors exceeds the number of observations, or when predictors are highly correlated, Elastic Net outperforms both Lasso and Ridge [32].  
Like Lasso and Ridge regression, Elastic Net starts with least squares. Then, it combines the Lasso regression penalty with the Ridge regression penalty. However, each gets its \(\lambda\), where \(\lambda\) is a regularization parameter controlling the regularization strength. For Ridge (L2), \(\lambda\) penalizes the sum of squared coefficients, and for Lasso (L1), it penalizes the sum of absolute coefficients. Optimal \(\lambda\) is selected through techniques like cross-validation [30]. Unlike Lasso regression, which tends to select a single correlated feature at random, the Elastic Net is more likely to select both features simultaneously [33].  
Scikit-learn offers a linear model named ElasticNet, which is trained with both L1 and L2 regularization to penalize the coefficients. This combination offers two advantages: it enables the development of a sparse model where few weights are non-zero, similar to Lasso regularization, while preserving the regularization properties of Ridge regularization.  
It minimizes the objective function:  
\[\frac{1}{2*n_{samples}}||\mathrm{y-Xw}||_{2}^{2}+\alpha*1I_{ratio}*||\mathrm{ w}||_{1}+0.5*\alpha*(1-1I_{ratio})*||\mathrm{w}||_{2}^{2}\]  
where:  
* \(\alpha\) is a constant that multiplies the penalty terms. Defaults to 1.0.
* \(I1_{ratio}\) is the ElasticNet mixing parameter, with \(0\leq 1I_{ratio}\leq 1\).
* The term \(\frac{1}{2*n_{samples}}\) normalizes the cost function based on the number of samples.
* \(||\mathrm{y-Xw}||_{2}^{2}\) measures the difference between predicted values (y) and actual values (Xw) using the L2 norm (squared Euclidean distance).
* \(\alpha*1I_{ratio}*||\mathrm{w}||_{1}\) encourages sparsity in the model weights (w) by penalizing their absolute values (L1 norm).
* \(0.5*\alpha*(1-1I_{ratio})*||\mathrm{w}||_{2}^{2}\) penalizes the model weights for magnitude using the L2 norm (squared values).  
The ElasticNet model (\(\alpha=0.1,1\!1\_\)ratio = 0.5) ranks feature importance by coefficient magnitude. Figure 3 shows the top 100 features by absolute coefficient values. Features on the left have larger coefficients, indicating stronger contributions to the model's predictions, while those on the right contribute less.