## 2 Background
### LASSO (Least Absolute Shrinkage and Selection Operator)  
The LASSO algorithm is a linear and logistic regression method that introduces an L1 penalty into the cost function, promoting sparsity of the solution. This penalty causes a subset of the coefficients associated with the features to be reduced exactly to zero, thus achieving automatic feature selection. In a classification context, the LASSO algorithm is used to identify the features that are most relevant for separating classes, reducing the dimensionality and noise in the data. This ability to perform automatic feature selection makes the LASSO algorithm a powerful and efficient method for classification [4].  
Consider a binary classification problem with a dataset \(X\in R^{nxp}\), where \(n\) is the number of observations, \(p\) is the number of features, and \(y\in\{0,1\}^{n}\) is the label vector. The linear model can be written as follows:  
\[\hat{y}=\mathrm{X}\beta\]  
where \(\beta=\left[\beta_{1},\ \beta_{2},\dots\ \beta_{p}\right]\) represents the coefficients associated with the features.  
In the standard mathematical formulation of the LASSO algorithm, the objective is to minimize the penalized cost function:  
\[\min_{\beta}\frac{1}{2n}\sum_{i=1}^{n}\left(y_{i}-X_{i}\beta\right)^{2}+ \lambda\sum_{j=1}^{p}\left|\beta_{j}\right|\]  
where the term \(\frac{1}{2n}\sum_{i=1}^{n}(y_{i}-X_{i}\beta)^{2}\) represents the prediction error, the term \(\lambda\sum_{j=1}^{p}\left|\beta_{j}\right|\) is the L1 constraint that controls the sparsity of the model, and the constant \(\lambda\) is a hyperparameter that determines the intensity of the penalty. As \(\lambda\) increases, the smaller coefficients \(\beta\) are reduced to zero, which leads to the exclusion of their respective features from the model.  
The LASSO algorithm is used in classification through its combination with a logistic model, known as logistic regression with LASSO penalty. For a binary classification problem, the cost function becomes:  
\[\min_{\beta}-\frac{1}{n}\sum_{i=1}^{n}\left[y_{i}\log(h(X_{i}\beta))+(1-y_{i} )\log\ (1-h(X_{i}\beta))\right]+\lambda\sum_{j=1}^{p}\left|\beta_{j}\right|\]  
where \(h(z)=\frac{1}{1+e^{-z}}\) is the sigmoid function.  
Steps of the LASSO algorithm for feature selection:  
* Data standardization: the X traits are scaled to have zero mean and unit standard deviation, preventing the favoring of traits with large values;
* Setting the hyperparameter \(\lambda\): cross-validation can be used to choose the optimal value of \(\lambda\) that balances the prediction error and sparsity: \(\circ\) A large \(\lambda\) removes more features, favoring sparse models; \(\circ\) A small \(\lambda\) includes more features, but may introduce noise;\(\circ\) The optimal value of \(\lambda\) is often selected by cross-validation;
* Model training: the penalized cost function is minimized using numerical methods, such as coordinate descent or gradient descent, which iteratively adjust the \(\beta\) coefficients;
* Removal of irrelevant features: after training, the features with coefficients \(\beta\)j = 0 are automatically removed, leaving only the relevant subset;
* Performance evaluation: the model is evaluated on a test dataset to verify its accuracy and generalization ability.  
It should be noted that the LASSO algorithm has both advantages and disadvantages in its application for feature selection. Thus, in the chapter on advantages, it should be noted that the algorithm can reduce the number of features without using separate preprocessing methods, that it integrates feature selection into the training of the model, and that it is robust in dealing with noisy data. The less strong points of the method are its poor performance in the presence of highly correlated features and the need for precise adjustment of the hyperparameter \(\lambda\)[21].