# Sparse Logistic Regression: Comparison of Regularization and Bayesian Implementations
## 4 Model Setting
### Monte Carlo Cross-Validation  
The Monte Carlo Cross-Validation (MCCV) procedure, which is also called repeated random sub-sampling validation [48] or repeated trining-test split or leave-group-out cross-validation in literature [49], as described in Algorithm 1 below, performs an analysis on \(K_{1}\) different random splits of the available observations into training/test datasets. Thus, \(K_{1}\) different models are built and the performance of the models is assessed as the average model performance over \(K_{1}\) simulations. For consistent results, \(K_{1}\) needs to be a large number (of the order of hundreds/thousands). The number of repetitions is important. Increasing the number of subsets has the effect of decreasing the uncertainty of the performance estimates. This is also a function of the proportion of samples were randomly allocated to the prediction set; the larger the percentage, the more repetitions are needed to reduce the uncertainty in the performance estimates [49].  
Moreover, in a classification problem, the dataset can be unbalanced, with one class having a much lower number of instances than the other one. To handle this problem, an undersampling strategy can be used where at each Monte Carlo (MC) iteration a new dataset is built randomly undersampling the most numerous class. After balancing the dataset, the choice is to split it into training and test part; these parts may not be balanced, so another choice could be to insert a stratified procedure to guarantee proportions of the two classes are maintained [50].  
A K-fold Cross-Validation (CV) procedure estimates the LASSO regularization parameter \(\lambda\) at step 6 of Algorithm 1. This procedure involves splitting the training set into \(K\) approximately equal parts. Each part is then selected in turn as a validation dataset and the remaining \(K-1\) parts used to estimate the model, which is evaluated on the validation dataset. The CV error is defined as the average of the misclassification error (MCE) (see Section 4.2) achieved on the \(K\) validation sets. The CV error is computed for a range of \(\lambda\) values and the value that yields the minimum CV error is selected as the optimum.  
Step 6 in Algorithm 1 is repeated \(K_{2}\) times and the median among the \(K_{2}\) complexity values is used to estimate the model during the \(i\)-th MC iteration. One difference between MCCV and k-fold cross-validation is that samples can be represented in multiple held-out subsets. Furthermore, the number of repetitions is usually larger than in k-fold cross-validation i.e., \(K_{1}>K_{2}\); in this case a choice of \(K_{1}=1000\) and \(K_{2}=10\) was made.  
```
Data: Input matrix \(X\), class labels vector \(Y\) Result: Performance Metrics: MCE, AUC,...
1 Set \(K_{1}\) and \(K_{2}\) ;
2 for\(i=1\)to\(K_{1}\)do
3 Build a balanced dataset, randomly undersampling the most numerous class;
4 Split the balanced dataset into training and test datasets;
5 for\(j=1\)to\(K_{2}\)do
6 Cross-Validation to find \(\lambda\) (LASSO);
7 Choose the complexity parameter as the median among the \(K_{2}\);
8 Identify models using the training dataset;
9 Test algorithms on test dataset;
```  
**Algorithm 1**Monte Carlo Cross-Validation algorithm.