# Variable Selection Using Deep Variational Information Bottleneck with Drop-Out-One Loss
## 4 Experimental Analysis
### Simulated Experiments
An empirical reason may be that the Gaussian distribution can generate a larger range on \(Z\) compared to the exponential distribution, providing the neural  
\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline  & & \multicolumn{3}{c}{\(\eta=0\)} & \multicolumn{3}{c}{\(\eta=0.2\)} & \multicolumn{3}{c}{\(\eta=0.5\)} \\ \cline{3-11} \((N,p)\) & **Models** & **TP** & **FP** & **STP** & **TP** & **FP** & **STP** & **TP** & **FP** & **STP** \\ \hline \multirow{8}{*}{\((100,10)\)} & Lasso [1] & 1.80 & 0.20 & 0.50 & 1.85 & 0.15 & 0.35 & 1.85 & 0.15 & 0.47 \\  & Elastic Net [32] & 1.95 & 0.05 & 0.21 & 1.90 & 0.10 & 0.30 & 1.90 & 0.10 & 0.30 \\  & SCAD [3] & **2.00** & **0.00** & **0.00** & 1.95 & 0.05 & 0.21 & 1.95 & 0.05 & 0.21 \\  & RF [13] & **2.00** & **0.00** & **0.00** & **2.00** & **0.00** & **0.00** & **2.00** & **0.00** & **0.00** \\  & AdaBoost [33] & 1.95 & 0.05 & 0.20 & 1.90 & 0.10 & 0.30 & 1.90 & 0.10 & 0.30 \\  & SpAM [4] & **2.00** & **0.00** & **0.00** & **2.00** & **0.00** & **0.00** & **2.00** & **0.00** & **0.00** \\  & GL [7] & **2.00** & **0.00** & **0.00** & **2.00** & **0.00** & **0.00** & 1.95 & 0.05 & 0.20 \\  & Knockoff [11] & 1.90 & 0.10 & 0.43 & 1.95 & 0.05 & 0.20 & 1.90 & 0.10 & 0.43 \\  & Lasso Net [14] & **2.00** & **0.00** & **0.00** & **2.00** & **0.00** & **0.00** & **2.00** & **0.00** & **0.00** \\  & DeepVIB (G) & **2.00** & **0.00** & **0.00** & **2.00** & **0.00** & **0.00** & **2.00** & **0.00** & **0.00** \\  & DeepVIB (E) & **2.00** & **0.00** & **0.00** & **2.00** & **0.00** & **0.00** & **2.00** & **0.00** & **0.00** \\ \hline \multirow{8}{*}{\((100,20)\)} & Lasso [1] & 1.80 & 0.20 & 0.40 & 1.75 & 0.25 & 0.62 & 1.15 & 0.85 & 0.96 \\  & Elastic Net [32] & 1.90 & 0.10 & 0.43 & 1.90 & 0.10 & 0.43 & 1.90 & 0.10 & 0.43 \\  & SCAD [3] & 1.90 & 0.10 & 0.30 & 1.90 & 0.10 & 0.43 & 1.80 & 0.20 & 0.50 \\  & RF [13] & 2.00 & 0.00 & 0.00 & 1.95 & 0.05 & 0.21 & 1.85 & 0.15 & 0.47 \\  & AdaBoost [33] & 1.95 & 0.05 & 0.21 & 1.85 & 0.15 & 0.47 & 1.75 & 0.25 & 0.62 \\  & SpAM [4] & **2.00** & **0.00** & **0.00** & **2.00** & **0.00** & **0.00** & **2.00** & **0.00** & **0.00** \\  & GL [7] & **2.00** & **0.00** & **0.00** & **2.00** & **0.00** & **0.00** & **2.00** & **0.00** & **0.00** \\  & Knockoff [11] & 1.85 & 0.15 & 0.35 & 1.85 network more choices when mapping \(Y\) from \(Z\). Hence, we prefer the Gaussian distribution when estimating the \(KL(p(z|x)|r(z))\) in DeepVIB. (4) As for other classical variable-selection algorithms, the experimental results rely on their properties. Methods, such as Elastic Net and SCAD, are proposed for linear models, and their performance is generally inferior to methods for nonlinear models, such as Lasso Net and RF, in nonlinear examples. We also note that Lasso Net performed perfectly in Example 4 but not in Example 3. The reason may be that the input data are drawn from different distributions and that the target function in Example 3 is more complex than the one in Example 4. In addition, we can see that SpAM performs well except in Example 2. An underlying reason is that SpAM was originally proposed with an additional assumption on the target function for sparse variable selection. In fact, Example 2 contains \(x_{2}\) in the denominator, which is not coincident with the additive assumption. In contrast, Examples 1 and 4 can be more like an additive model except for the interaction term between \(x_{1}\) and \(x_{2}\). Different from SpAM, Lasso Net was proposed based on DNNs and, thus, does not rely on a specific assumption of the target function. The strong representation and approximation ability of DNNs ensure that Lasso Net can achieve satisfactory performance on the four examples. This also supports our motivation to investigate DeepVIB for variable selection. To highlight the influence of parameter \(\beta\) in the proposed method, we define a new static by empirical statistics:  
\[gap=\min_{k\in T}\hat{S}_{k}-\max_{j\in T}\hat{S}_{j},\]  
where a large \(gap\) means better selection results.