## 2 Materials and Methods
### Machine Learning Regression
#### 2.3.1 Ridge Regression (RR), Lasso Regression (LR), and Elastic Net Regression (ENR)  
In the course of training and evaluating machine learning models, often the error resulting from either the training or testing dataset or both exceeds the anticipated level, which causes overfitting. A model is said to be overfitted when it performs exceptionally well on the training set but fails to generalize effectively to the unseen instances, leading to elevated error values. This could probably happen due to several reasons, one of which is the bias-variance trade off. Oversimplified models characterize high bias, while the inability of a model to generalize to unseen data signifies high variance. As bias is reduced, the model becomes more vulnerable to high variance and, conversely, when variance is decreased, bias may increase. Therefore, achieving a good predictive model necessitates striking a trade off or balance between these two measures. Regularization is an approach that addresses this issue and offers better performance in handling multicollinearity and overfitting. In regularization, the least-squares regression cost function '\(L_{ls}\)' (Equation (4)) is modified by adding a penalty term that penalizes large model coefficients and increases the generalizability. In the cost function, \(\beta_{0}\) is the model coefficient or intercept, \(\hat{\beta}\) is the estimate of the true parameters, and \(y_{i}\) is the actual value of the target variable calculated on \(x_{i}\) from a dataset containing 'n' instances. \[L_{ls}=\underset{\hat{\beta}}{\text{argmin}}\Bigg{\{}\sum_{i=1}^{n}\bigl{(}y_{ i}-\beta_{0}-x_{i}\hat{\beta}\bigr{)}^{2}\Bigg{\}} \tag{4}\]Popular regularization techniques in regression include Ridge Regression (RR), Lasso (Least Absolute Shrinkage and Selection Operator) Regression (LR), and Elastic Net Regression (ENR). RR is characterized by a penalty term, which incorporates half the square of the 'L2' norm and is added to the cost function [58; 59]. The L2 norm encompasses the sum of squared differences between predicted and target values over the feature vector and this formulation imposes a constraint on the learning algorithm, compelling it not only to fit the data but also to minimize the model weights. The amount of ridge regularization is determined by the shrinkage hyperparameter '\(\lambda\)', and its proper selection is crucial. When \(\lambda\) is equal to zero the RR becomes equivalent to linear regression. Conversely, when \(\lambda\) is exceedingly large, all the weight values converge to nearly zero, leading to an underfitted model. The cost function of LR is as shown in Equation (5). \[L_{rr}=\underset{\hat{\beta}}{\text{argmin}}\left\{\sum_{i=1}^{n}\left(y_{i}- \beta_{0}-x_{i}\hat{\beta}\right)^{2}+\lambda\sum_{j=1}^{m}\hat{\beta_{j}}^{2}\right\} \tag{5}\]  
On the other hand, instead of half the square of L2, the LR uses 'L1' norm as penalty term in the cost function. L1 norm is the sum of the absolute values of the difference between actual and predicted target values over the feature vector and is also known as the Manhattan distance (Equation (6)). In LR, \(\lambda\) operates similar to that in RR but a fundamental distinction lies in the treatment of coefficients. While both the techniques shrink the coefficients towards zero, LR is unique in that it assigns them exactly to zero when shrinkage reaches a certain threshold. Consequently, this leads to the creation of a model with a selected subset of features, rendering it considerably more interpretable and manageable in practice, particularly for datasets with several input features [58; 60; 61]. \[L_{lr}=\underset{\hat{\beta}}{\text{argmin}}\left\{\sum_{i=1}^{n}\left(y_{i}- \beta_{0}-x_{i}\hat{\beta}\right)^{2}+\lambda\sum_{j=1}^{m}|\hat{\beta_{j}}|\right\} \tag{6}\]  
In contrast, ENR can be considered as a hybrid approach, combining characteristics from RR and LR. The penalty term in ENR comprises a blend of absolute-value and squared-value penalties.