# Sparse Logistic Regression: Comparison of Regularization and Bayesian Implementations
## 6 Results and Discussion
### Simulated/Artificial Data
With regard to LASSO, in Table 2 it can be noticed that for increasing values of \(n\), in general, the IC value improves, i.e., is lower, even though condition (17) is not satisfied for several cases in which the LASSO structure recovery capability fails. It is pointed out that several approaches exist in the literature as alternatives to LASSO with regard to model recoveries, such as the smoothly clipped absolute deviation (SCAD) and the adaptive lasso [29], but they are not used here in order to keep the focus on the two methods mentioned: LASSO and RVM, whose variable selection capabilities are embedded within the logistic model estimation phase. The structure discovery ability of the two techniques is depicted in Figure 4 for dataset (f), where the true vector \(\theta\) (red circle) is compared with the normalized distribution obtained with the MC simulation with LASSO (blue square) and RVM (green diamond) for increasing values of \(n\) (from top left to bottom right). Figure 4 can be used to appreciate the capability of each of the two techniques in terms of structure recovery in a specific case study (dataset (f)). In this case, the predictors with more impact on the model are known in advance and are depicted in red (circle) in the figure. In this scenario, one method is better than the other if its estimates tend to overlap the real coefficient, i.e., if the stem plots of the real coefficients and estimates show nonzero values for the same indices. It can be seen that, neither RVM, nor LASSO are able to reveal the correct predictors when \(n\) is small, instead, as \(n\) increases, both of the methods tend to provide more reliable coefficients as expected. The important fact observed and pointed out here is that RVM seems to be more powerful for model recovery than LASSO when \(n\) is large, because when it shows nonzero coefficients, they match the real ones. In order to improve the readability of the picture, the reader can consider it together with Table 3 and Figure 8a which show the same conclusions for the same data quantifying the differences. In particular, Figure 8a represents how the number of selected (nonzero) variables tends to be lower for RVM as \(n\) increases. On the other hand, Table 3 shows the matches between the selected variables and  
\begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline \hline
**Dataset** & \multicolumn{4}{c}{**(a)**} & \multicolumn{4}{c}{**(b)**} & \multicolumn{4}{c}{**(c)**} \\ \hline p & \multicolumn{2}{c}{100} & \multicolumn{2}{c}{1000} & \multicolumn{2}{c}{1000} & \multicolumn{2}{c}{1000} & \multicolumn{2}{c}{1000} & \multicolumn{2}{c}{1000} & \multicolumn{2}{c}{1000} \\ \hline n & 100 & 1000 & 100 & 1000 & 200 & 1000 & 200 & 1000 & 200 & 1000 & 200 & 1000 & \\ \hline IC & 1.236 & 0.347 & 1.575 & 0.426 & 1.023 & 1.004 & 1.072 & 1.03 & 0.851 & 0.357 & 1.113 & 0.471 & \\ \hline
**Dataset** & \multicolumn{4}{c}{**(d)**} & \multicolumn{4}{c}{**(f)**} & \multicolumn{4}{c}{**(f)**} \\ \hline p & \multicolumn{2}{c}{100} & \multicolumn{2}{c}{1000} & \multicolumn{2}{c}{80} & \multicolumn{2}{c}{120} & \multicolumn{2}{c}{200} & \multicolumn{2}{c}{280} & \multicolumn{2}{c}{400} & \multicolumn{2}{c}{800} & \multicolumn{2}{c}{1200} & \multicolumn{2}{c}{1600} & \multicolumn{2}{c}{2000} \\ \hline n & 200 & 1000 & 200 & 1000 & 100 & 1000 & 1000 & 1000 & 1000 & 1000 & 1000 & 1000 & 1000 \\ \hline IC & 0.75 & 0.446 & 0.972 & 0.486 & 2.144 & 2.155 & 2.025 & 2.092 & 2.129 & 2.112 & 2.093 & 2.092 & 2.0 the real ones in percentage for each method. RVM appears more effective for model recovery in this case study; nevertheless, it is noteworthy to consider that this dataset presents high collinearity among variables, as already stated, and LASSO suffers somewhat from the multi-collinearity problem [33]. It turns out that, for increasing values of \(n\), the RVM performs better than LASSO identifying the true \(\theta_{j}\
eq 0\), namely those that are used to predict the output class.