# Sparse Logistic Regression: Comparison of Regularization and Bayesian Implementations
## 6 Results and Discussion
### Computational Considerations  
Both implementations of the linear logistic model considered in this work are based on iterative algorithms. Thus, the time required for model parameter estimation depends on thresholds and the number of iterations specified by the user. The update of parameters in the RVM implementation requires the inversion of a matrix to compute the posterior weight covariance matrix, an operation with complexity \(\mathcal{O}(p^{3})\) and memory requirement \(\mathcal{O}(p^{2})\) ([33; 42]). The algorithm implementation described in [44] does not start by including all variables in the model but rather incrementally adds or deletes variables as appropriate. Moreover, as already noted, complexity is automatically determined. The LASSO algorithm consists of several steps. The coordinate descent algorithm requires \(\mathcal{O}(pn)\) ([33]), but additional complexity is due to the calculation of the quadratic approximation by the Newton algorithm. An example of comparison in terms of execution time performance has been provided in Figure 10 for 2 real cases introduced before: Arcene, and Fabric Care Appliance. These two case studies have been chosen in this case among all the others because of their \(\frac{n}{p}\) ratio which is very different from one to other. Figure 10 shows distributions of the execution times obtained for each MCCV iteration for each method. The elapsed time for each execution was measured by using tic-toc command available in MATLAB. This couple of instructions was used to tag the beginning and the end of the key code for each method so that the difference in time between the two tags indicates the desired execution time. More specifically, the tic function records the current time, and thetoc function uses the recorded value to calculate the elapsed time (source: [[https://www.mathworks.com/help/matlab/ref/tic.html](https://www.mathworks.com/help/matlab/ref/tic.html)]([https://www.mathworks.com/help/matlab/ref/tic.html](https://www.mathworks.com/help/matlab/ref/tic.html))). The main steps related to the implementation of the two procedures have been summarized in Table 5 for the sake of clarity. These steps affect the execution time and are useful to stress the methodological differences between the two strategies as well. As the reader can notice, only LASSO performs the choice of the optimal regularization parameter with a dedicated inner procedure that turns out to be particularly demanding considering the entire set of iterations. On the other hand, RVM determines the model complexity automatically by the data through ARD as already stated in Section 3.2. Step I in Table 5 refers to the code used in the glmnet LASSO implementation to determine the optimal regularization parameter lambda at a grid of values via Coordinate Descent algorithm. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \multicolumn{2}{c}{**MCE**} & \multicolumn{2}{c}{**AUC**} \\ \cline{2-5}  & **LASSO** & **RVM** & **LASSO** & **RVM** \\ \hline SAheart & 0.32 \(\pm\) 0.05 & 0.30 \(\pm\) 0.04 & 0.75 \(\pm\) 0.05 & 0.77 \(\pm\) 0.04 \\ Abalone & 0.28 \(\pm\) 0.12 & 0.24 \(\pm\) 0.11 & 0.81 \(\pm\) 0.12 & 0.82 \(\pm\) 0.12 \\ Semiconductor & 0.24 \(\pm\) 0.02 & 0.21 \(\pm\) 0.02 & 0.82 \(\pm\) 0.05 & 0.87 \(\pm\) 0.02 \\ Arcene & 0.26 \(\pm\) 0.06 & 0.24 \(\pm\) 0.06 & 0.82 \(\pm\) 0.06 & 0.81 \(\pm\) 0.06 \\ Madelon & 0.38 \(\pm\) 0.01 & 0.39 \(\pm\) 0.01 & 0.63 \(\pm\) 0.01 & 0.64 \(\pm\) 0.01 \\ Gisette & 0.04 \(\pm\) 0.00 & 0.07 \(\pm\) 0.07 & 0.99 \(\pm\) 0.00 & 0.96 \(\pm\) 0.08 \\ Fabric Care Appliance & 0.03 \(\pm\) 0.02 & 0.04 \(\pm\) 0.02 & 0.99 \(\pm\) 0.01 & 0.98 \(\pm\) 0.01 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Mean and standard deviation of the MCE and the AUC distributions obtained from the MC analysis for real datasets. The lower the MCE and the higher the AUC values the better in term of accuracy.