# Sparse Logistic Regression: Comparison of Regularization and Bayesian Implementations
## 6 Results and Discussion  
### Simulated/Artificial Data  
The results for the simulated datasets are presented Figures 3-8. In particular, Figures 4-8 refer to dataset (f)--Manufacturing, while Figure 3 refers to other datasets (a)-(e). As expected, in general, the performance for all indicators improves for both methods with increasing values of \(n\), for example the MCE in Figure 5 for dataset (f) decreases as \(n\) increases. Moreover, Figure 3 shows that, in general, for the same value of \(n\), the lower the value of \(p\) the better the performance. RVM outperforms LASSO when \(p\leq n\) in terms of MCE and AUC. Increasing Prediction Accuracy (PA) is obtained for both methods when \(p\) and \(n\) are comparable with respect to the case \(p>>n\). For example, Figure 3 shows a drop for simulation (a) in terms of average MCE from 30% to 5% for LASSO and from 25% to 3% for RVM when \(n=100\) and \(n=1000\) respectively. Similar performance improvements are obtained for simulations (b)-(d) as can be seen from Figure 3. Additionally, the same figure shows that RVM tends to outperform LASSO in the estimation of the true model structure (or Model Recovery--MR) in terms of Sensitivity, i.e., True Positive rate, and Specificity, i.e., True Negative rate. The set of simulations (e), basically the same as (a)-(d) but repeated for \(p=1000\), whose results are also shown in Figure 3, confirm that RVM tends to outperform LASSO in all situations, although in a few cases results are comparable, especially when \(p>>n\). In this situation (\(p>>n\)), the structure discovery ability of the RVM degenerates. This can be due to the fact that the hyperparameters for RVM are optimized with an iterative procedure (see Section 3.2) that does not guarantee the cost function minimum is reached. On the other hand, the LASSO problem is convex and thus the coordinate descent algorithm is more likely to reach the minimum of the cost function. Several studies have been conducted in the literature studying the selection consistency property of the LASSO estimator. A necessary condition for the LASSO to recover the true model structure is the so-called Irrepresentable Condition (IC) [54; 55] calculated as:  
\[\underset{k\in N}{max}|sign(\mathbf{\theta}_{S})^{T}(\mathbf{X}_{S}^{T}\mathbf{T }_{S})^{-1}\mathbf{X}_{S}^{T}\mathbf{X}_{k}|<1 \tag{17}\]  
where the subscript \"\(S\)\" indicated the set of variables whose coefficients are different from zero, while \(N\) identifies the \(k\) variables with \(\theta_{k}=0\). If this condition is violated, the recovery of the true model is not guaranteed. Table 2 collects the IC values obtained for the simulated datasets (a)-(f). Interestingly, observing Figure 3 and Table 2 it is possible to notice that the lower the IC, the higher the Specificity values in model recovery (MR) for LASSO, and specificity values are also higher for RVM when \(n\) is large. It can be also noticed that the irrepresentable condition (17) occasionally fails when the IC value is greater than 1; in these cases, the structure recovery provided by LASSO is not suitable according to (17). Figure 3: Summary of the simulation results obtained with simulated datasets **(a–d)**: DLASSO results; RVM results. Figure 4: Coefficients estimated with LASSO (blue) and RVM (green) compared with the real ones (red) for dataset (f). Figure 5: dataset (f): MCE (**a**) and AUC (**b**) for LASSO (dashed) and RVM (solid). The average MCE and AUC values are represented by tick lines, while the shaded areas represent the standard deviation obtained across the MC simulation. Figure 6: dataset (f): Sensitivity (**a**) and specificity (**b**) as regard **Prediction Accuracy (PA)** for LASSO (dashed) and RVM (solid). The shaded areas represent the standard deviation obtained across the MC simulation. Figure 7: dataset (f): Sensitivity (**a**) and Specificity (**b**) as regard **Model Recovery (MR)** ability for LASSO (dashed) and RVM (solid). Shaded areas for standard deviation across the MC simulations.