# Sparse Logistic Regression: Comparison of Regularization and Bayesian Implementations
## 7 Conclusions  
Increasingly high dimensional datasets are encountered in different fields of science and engineering. To handle such datasets effectively techniques are required to integrate variable selection with model building capabilities in a robust and efficient manner. In particular, it is often interesting to identify the key variables driving output variation. This work is focused on a comparison between two sparse techniques that are well established in the literature but which have not, to date, been subject to a comprehensive comparison on several datasets with different characteristics; therefore, the aim here was to provide a reference and a tutorial for such a comparison in terms of implementation differences, performance and structure recovery. The first method, the LASSO, encodes the sparsity in the cost function and is based on regularization. The second method, RVM, encodes sparsity by setting priors over model coefficients and is based on a Bayesian implementation of the logistic model. As regards prediction accuracy and for each setting considered in the simulation study, results showed that:  
1. in general RVM and LASSO appear equivalent in terms of prediction accuracy. Experimental evidence shows that RVM turns out to be better as \(n\) increases: \(\frac{n}{p}\) is far from 1;
2. when data is known to be strongly non-linear, as for example is the case with the Madelon dataset, LASSO shows comparable or slightly better prediction accuracy; this conclusion should be confirmed by investigating other similar highly non-linear datasets. Concerning model recovery:  
* when \(p\leq n\) RVM tends to outperform the LASSO;
* when \(p>>n\) the situation changes. In this case, the smaller the \(n/p\) ratio becomes, the more LASSO seems to identify the true underlying structure. Figure 10: Examples of distributions of execution times in seconds for RVM and LASSO depending on MCCV iterations (1000). Arcene data in (**a**), and Fabric Care Appliance data (**b**). For the latter case, the reason can be due to the approximation procedure used to estimate the unknown quantities, i.e., model parameters, and hyperparameters, of the RVM model which suffers from large values of \(p\). As explained in Section 3.2, the RVM implementation of the logistic model resembles an Expectation-Maximization like algorithm ([33]), where all the unknowns are iteratively updated. Large values of \(p\) make such an algorithm more sensitive to local minima of the cost function leading to possible suboptimal model parameter estimation. Considering the experimental evidence of this study, our recommendation in accord with other results from similar works in the literature (such as [40]), is the use of LASSO when the available dataset is not sufficiently large or there is no a priori knowledge on its structure; on such occasions RVM could provide unstable predictions. Both for prediction performances and structure recovery, the choice for the most suitable sparse learner depends on the case highlighted by the ratio \(\frac{n}{p}\) as reported above. As a final observation, RVM does not require the estimation of a regularization parameter. This could be an advantage since a completely data-driven approach is less computationally expensive and less user-subjective. On the other hand, the manual setting of the complexity parameter for the LASSO can be useful when there is a need to tailor the model complexity to the specific problem under analysis. Despite the widespread popularity of the LASSO algorithm in the last decade, with this work, we have shown that RVM deserves more attention as a candidate technique for the estimation of sparse models. Not only it is competitive in terms of classification prediction accuracy and model recovery when compared to the LASSO, but it also does not have the overhead of a cross-validation procedure for complexity parameter tuning. Conceptualization, M.Z., G.A.S. and S.M. ; methodology, M.Z., G.A.S. and S.M. ; software, M.Z., G.A.S. and G.Z. ; validation, M.Z., G.Z. ; formal analysis, S.M. ; G.A.S.