# Sparse Logistic Regression: Comparison of Regularization and Bayesian Implementations
## 6 Results and Discussion
### Simulated/Artificial Data
Considering Table 2, the reader can notice that the LASSO Irrepresentable Condition fails also for increasing values of \(n\), so it is not surprising that better results for model recovery, in this case, are obtained with the other technique. Figure 5 shows results in terms of MCE (left) and AUC (right) for dataset (f). They are presented as a function of the number of samples \(n\) used to build the dataset at each MC iteration. The RVM models (solid line) outperform the LASSO one (dashed line) as \(n\) increases to a value approximately equal to \(p\). For low values of \(n\), i.e., when \(p>>n\), results for the LASSO and RVM became comparable as happened for simulations (b) in Figure 3. However, analyzing Figure 6 for dataset (f), one can see that RVM still outperforms the LASSO in terms of sensitivity (left) and specificity (right) for prediction accuracy (PA) considering the predicted output for the two models during the MC simulation. Figure 7 shows for dataset (f) the TP (or sensitivity) and TN (or specificity) rates for the model recovery capability of the two studied models. Not only LASSO often selects the wrong variables, but the selected variables are spread around the true ones (observing Figure 7 together with Figure 4). This is an example of the so-called grouping effect affecting the LASSO, namely, if there is a group of highly correlated variables, LASSO tends to select only one of them from the group and does not care which one is selected. The data under analysis are actually highly correlated in some occasions as can be seen from Figure 2 which shows the heat-map matrix of the absolute correlations between the OES dataset variables (dataset (f)). As can be seen from Figure 8 for dataset (f), while for the lower \(n\) RVM selects fewer variables than LASSO, the situation changes for increasing values of \(n\). Despite this behavior in the sparsity of the two models, the RVM model still outperforms the LASSO in terms of MCE (continuous decreasing line in Figure 8 compared with dashed decreasing line). In particular, Figure 8 for dataset (f) shows that for an equal number of active variables in the two models, RVM achieves better prediction performance. somehow, the non-linear structure of data is known in advance for the problem at hand; however, this consideration should be supported by other similar cases;
* The Gisette dataset is very noisy and this is due to the intrinsic difficulty of separating the digits \"4\" and \"9\"; also in this case LASSO turns out to be better than RVM;
* For the Arcene dataset where \(p>>n\), RVM slightly outperforms LASSO in terms of MCE, but LASSO is better in terms of AUC. Concerning the structure recovery of the two algorithms using real datasets, some examples are depicted in Figure 9. With the Semiconductor dataset in Figure 9a RVM gives a distribution of nonzero variables in cross-validations which is less than the one provided by LASSO (for example the mean is 2.69 for RVM and 4.99 for LASSO). The same situation, applies for the Arcene dataset (Figure 9b) where \(p>>n\), i.e., 14.63 mean for RVM and 16.81 for LASSO. In contrast, for the highly nonlinear dataset Madelon in Figure 9c, the difference between the mean number of selected variables in cross-validations is much higher with RVM than with LASSO (179.58 versus 26.92). As a result, the use of LASSO can be recommended in general when dataset is non-linear (or even more when there is no a priori knowledge about the structure). For real data case the same set of implementation details was kept, i.e., in terms of number of Cross-Validation, percentage of training and test data, etc. (as explained in Section 6.1). Figure 9: Examples of number of selected variables for RVM and LASSO depending on MCCV iterations (1000). Semiconductor data in (**a**), Arcene (**b**) and Madelon (**c**).