# Sparse Logistic Regression: Comparison of Regularization and Bayesian Implementations
## 5 Datasets
### Simulated/Artificial Data  
The simulated datasets are generated by imposing a certain correlation structure on the input matrix \(\mathbf{X}\):  
1. Independent predictor variables: all \(p=100\) predictor variables are IID standard normal distributed; sample size \(n=100\) and \(n=1000\); \begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Observations** & **Variables** & **Samples Distribution** \\  & **(_n_)** & **(_p_)** & **(_k_ = 0,1) [Ratio_]** \\ \hline SA heart & 462 & 9 & 160/302 \\ Abalone & 4174 & 8 & 32/4132 \\ Semiconductor & 2194 & 1988 & 410/1784 \\ Arcene & 200 & 10,000 & 88/112 \\ Madelon & 2600 & 500 & 1300/1300 \\ Gisette & 7000 & 5000 & 3500/3500 \\ Fabric Care Appliance & 304 & 52 & 146/158 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Real datasets summary. The last column indicates the number of observations belonging to each of two classes. Madelon is an artificial dataset and included here among real data. Figure 1: Diversity of real/artificial datasets in terms of class samples distribution and \(\frac{n}{p}\) ratio. 2. Toeplitz design: the \(p=100\)-dimensional predictor variables follow an \(\mathcal{N}(0,\mathbf{\Sigma})\) distribution, where \(\mathbf{\Sigma}=\rho^{|k-m|}\) and \(\rho=0.99\); sample size \(n=200\) and \(n=1000;k,m=1,\dots,p\). Therefore \(\Sigma\) is a matrix of p rows and p columns made up by all combinations of \(\rho\), \(k\), \(m\),;
3. Factor model with two factors: let \(\phi_{1}\) and \(\phi_{2}\) be two latent variables following IID standard normal distributions. Each predictor variable \(x_{k}\), for \(k=1,\dots,p\), is generated as a combination of gaussians \(x_{k}=f_{k,1}\phi_{1}+f_{k,2}\phi_{2}+\eta_{k}\), where \(f_{k,1},f_{k,2}\) and \(\eta_{k}\) have IID standard normal distributions for all \(k=1,\dots,p\); sample sizes are \(n=200\) and \(n=1000\) while \(p=100\);
4. Data set (d) is identical to (c) but with 10 instead of two factors;
5. All simulations (a)-(d) are repeated with \(p=1000\). 6. Manufacturing dataset: as shown in Figure 2, this dataset shows high block-correlation among subsets of input variables and has \(p=1988\) and \(n=2194\) (the Pearson correlation coefficient was used for the computation [51]). The variables represent statistical moments (Mean, Variance, Skewness and Kurtosis), calculated from Optical Emission Spectroscopy (OES) data from a plasma etching process. Independent variables of this dataset are used in this paper to obtain a first case study for simulated/artificial environment (constructing a simulated output) in order to study prediction accuracy and model recovery for different \((n,p)\). In this case we know which variables are truly relevant. A second case study for real environment (assuming a specific measured output as target). For this reason the same independent variables are also used and explained better in the next Section 5.2. In this subsection we do not use the response values from the real data set, however, we need to know which variables are truly relevant or irrelevant. For this reason for all the above cases we proceed as follows. * Having defined each of the synthetic \(\mathbf{X}\), the simulated output is obtained with the logistic linear model assuming a binomial distribution;
* We create sparse regression vectors by setting \(\theta_{j}=0\) for all \(j=1,\dots,p\) except for a randomly chosen set \(S\) of coefficients, where \(\theta_{j}\) is chosen independently and uniformly in \([0,1]\) for all \(j\in S\). The size \(s=|S|\) of the active set chosen as 20. Another dataset considered in this section is called **Madelon**; it is actually an artificial dataset containing data points grouped in 32 clusters placed on the vertices of a five-dimensional hypercube and randomly labeled \(+1\) or \(-1\). It is highly non-linear and it presents many correlated variables.