# Sparse Logistic Regression: Comparison of Regularization and Bayesian Implementations
## 3 Methods
### Relevance Vector Machine (RVM)  
Here, the linear logistic model, whose likelihood is described by (3), is formulated within a Bayesian framework where sparseness is encoded by setting priors on the model coefficients \(\theta_{j}\). This a priori information, i.e., the knowledge, before looking at the data, that all variables are important, is updated during the learning process where evidence from the data is used to set \(\theta_{j}\) corresponding to non-infomative variables to zero. This implementation, called the Relevance Vector Machine [15], estimates the posterior distribution of the parameter vector \(\mathbf{\theta}\) and the hyperparameters \(\mathbf{\alpha}\in\mathbb{R}^{p}\) governing sparseness (see below) over the data. Therefore, from Bayesian representation:  
\[p(\mathbf{\theta},\mathbf{\alpha}|\mathbf{Y})=\frac{p(\mathbf{Y}|\mathbf{\theta},\mathbf{ \alpha})p(\mathbf{\theta},\mathbf{\alpha})}{p(\mathbf{Y})} \tag{10}\]  
where \(p(\mathbf{Y}|\mathbf{\theta},\mathbf{\alpha})\) is the likelihood function and \(p(\mathbf{\theta},\mathbf{\alpha})\) is the prior distribution defined over the model coefficients and the hyperparameters, respectively. The hyperparameter vector \(\mathbf{\alpha}\) is encoded as a zero-mean Gaussian distribution over \(\mathbf{\theta}\) (popular simple choice discussed in [15]):  
\[p(\mathbf{\theta}|\mathbf{\alpha})=\prod_{j=1}^{p}\mathcal{N}\left(\theta_{j}|0, \alpha_{j}^{-1}\right) \tag{11}\]and a further set of hyperparameters Gamma distributed over the \(\mathbf{\alpha}\) vector is then introduced; for further details the interested reader is referred to [15]. Given a test data point \(\tilde{x}\), predictions are made for the corresponding target \(\tilde{y}\), in terms of the predictive distribution:  
\[p(\tilde{y}|\mathbf{Y})=\int p(\tilde{y}|\mathbf{\theta},\mathbf{\alpha})p(\mathbf{\theta},\mathbf{\alpha}|\mathbf{Y})d\mathbf{\theta}d\mathbf{\alpha} \tag{12}\]  
The posterior (10) cannot be computed because the normalizing integral on the right-hand-side \(p(\mathbf{Y})=\int p(\mathbf{Y}|\mathbf{\theta},\mathbf{\alpha})p(\mathbf{\theta},\mathbf{ \alpha})d\mathbf{\theta}d\mathbf{\alpha}\) cannot be obtained analytically. The posterior is thus decomposed as:  
\[p(\mathbf{\theta},\mathbf{\alpha}|\mathbf{Y})=p(\mathbf{\theta}|\mathbf{\alpha},\mathbf{Y})p( \mathbf{\alpha}|\mathbf{Y}). \tag{13}\]  
An iterative algorithm can then be used, where an estimate of \(\mathbf{\alpha}\) given by the second term on the right-hand side of (13) is used to calculate \(\mathbf{\theta}\); this is then used to update the previous estimation of \(\mathbf{\alpha}\). In particular, the posterior of \(\mathbf{\theta}\sim\mathcal{N}\left(\mathbf{\theta}^{RVM},\Sigma^{RVM}\right)\) in terms of the parameters of a Gaussian distribution \((\mathbf{\theta}^{RVM},\Sigma^{RVM})\) can be found by maximizing the first term on the right-hand-side of (13) [43]. The vector \(\mathbf{\theta}^{RVM}\) is obtained by maximizing the quantity \(p(\mathbf{\theta}|\mathbf{\alpha},\mathbf{Y})\propto p(\mathbf{Y}|\mathbf{\theta})p(\mathbf{ \theta}|\mathbf{\alpha})\) using a second order Newton gradient ascent method, once it is written in the following form discussed in [15]:  
\[log(p(\mathbf{Y}|\mathbf{\theta})p(\mathbf{\theta}|\mathbf{\alpha}))=l(\mathbf{\theta})-\frac{ 1}{2}\{\mathbf{\theta}^{T}\mathbf{A}\mathbf{\theta}\} \tag{14}\]  
where \(\mathbf{A}=diag(\alpha_{1},\ldots,\alpha_{p})\). Note that (14) is a regularized version of the log-likelihood function (4) and that there is a hyperparameter \(\alpha_{j}\) associated with each coefficient and thus each variable of the logistic model. A Laplace approximation is then used to calculate \(\Sigma^{RVM}\)[43].