# Stability of Feature Selection in Multi-Omics Data Analysis
## 2 Materials and Methods  
### Classifiers with Embedded Feature Selection  
In this study, we employed classifiers with embedded feature-selection mechanisms based on the L1 norm. The L1 norm, or Lasso regularization, is particularly effective in high-dimensional settings where the number of features exceeds the number of observations, as it encourages sparsity in the feature space. The classifiers used in our analysis include Support Vector Machine (SVM), Logistic Regression (LR), and Lasso regression, each of which integrates feature selection directly within the model-training process.  
Let the dataset consist of \(m\) observations \(O_{j}\) (\(j=1,...,m\)). Each observation \(O_{j}\) is described by \(n\) features and represented as an \(n\)-dimensional feature vector \(\mathbf{x}_{j}=[x_{j1},...,x_{jn}]^{T}\). The components \(x_{ji}\) of the vectors \(\mathbf{x}_{j}\) are referred to as features. Each feature vector \(\mathbf{x}_{j}\) is associated with a binary label (or class) \(y_{j}\). The positive class, indicating the presence of a disease, is typically denoted by 1, while the negative class can be represented as either 0 (as used in logistic and linear regression) or \(-1\) (as is common in Support Vector Machine models). Consequently, we can express the general-criterion function as follows:  
\[\Phi_{C}(\mathbf{w},\theta)=\frac{1}{m}\cdot\sum_{j=1}^{m}error(y_{j},\hat{y}( \mathbf{x}_{j},\mathbf{w},\theta))+\frac{1}{C}\sum_{i=1}^{n}|w_{i}|\, \tag{1}\]  
where \(\mathbf{w}=[w_{1},...,w_{n}]^{T}\in\mathbb{R}^{n}\) represents the weight vector, and \(\theta\in\mathbb{R}^{1}\) denotes the threshold of the hyperplane \(H(\mathbf{w},\theta)\). These parameters are determined during the minimization of the criterion function \(\Phi_{C}(\mathbf{w},\theta)\).  
The criterion function in Equation (1) consists of two parts. The first term represents the loss function, which measures the difference between the true values \(y_{j}\) and the predicted values \(\hat{y}(\mathbf{x}_{j},\mathbf{w},\theta)\). The second term is the regularization term, which penalizes the values of the model weights \(\mathbf{w}\).  
While the entire function needs to be minimized during training, the regularization term ensures that the model weights remain small. The hyperparameter \(C\) controls the relative importance of the two terms: a smaller value of \(C\) gives more weight to regularization (encouraging smaller weights), while a larger value of \(C\) emphasizes minimizing the loss function, allowing for higher weights. The value of \(C\) must be set prior to optimization and determines the trade-off between model fit and complexity.  
Using the absolute values (the L1 norm) of the coefficients \(|w_{i}|\) in the second regularization term has the property that during optimization, the weights not only become smaller but actually tend to become exactly zero, thus performing feature selection. In the case of the L2 norm, where squared weights are used in the regularization term \(w_{i}^{2}\), the weights tend to go towards zero but not exactly zero. This property of L1 regularization makes it useful for high-dimensional datasets.  
A lower value of \(C\) reduces the weight assigned to model errors while increasing the strength of regularization. This results in more weights being driven to zero, leading to a sparser model with fewer features. Conversely, a higher value of \(C\) increases model complexity by allowing the inclusion of more features, enabling the model to fit the training data more closely. This close fit is generally undesirable in the context of multi-omics data, where the number of features \(n\) significantly exceeds the number of observations \(m\).