# Variable Selection Using Deep Variational Information Bottleneck with Drop-Out-One Loss
## 4 Experimental Analysis
### Simulated Experiments  
For the simulated data, we consider the \(\epsilon\) in (4) following the Gaussian distribution \(\mathcal{N}(0,1)\) scaled by \(0.05\) and the same target functions \(f^{*}\) as adopted in previous papers [9; 34]. Among them, \(f^{*}\) is non-additive in Examples 1, 2 and 4 and is additive in Example 3. **Example 1**. : _The non-additive model (from [9])_  
\[f^{*}(\mathbf{x})=x_{1}x_{2}+(2x_{3}-0.5)^{2}+x_{4}+x_{5},\]  
_where \(X=[X_{1},X_{2},\cdots,X_{10}]\) are i.i.d. and drawn from the uniform distribution on \([0,1]\) with \(N=100\)._  
**Example 2**. : _The non-additive model (from [9])_  
\[f^{*}(\mathbf{x})=\frac{x_{1}}{0.5+(1.5+x_{2})^{2}}.\]  
_In this situation, \(X=[X_{1},X_{2},\cdots,X_{10}]\) are drawn from \(\mathcal{N}(0,\Sigma)\) with \(N=100\), where \(\Sigma\) denotes the covariance matrix with the \((i,j)\)th entry given by \(\Sigma_{i,j}=0.5^{|i-j|}\)._  
**Example 3**. : _The additive model (from [9])_  
\[f^{*}(\mathbf{x})=-2\tan(0.5x_{1})+x_{2}+x_{3}+\exp(-x_{4}).\]  
_Here, we consider three cases \((N,p)=(100,10),(100,20)\) and \((200,50)\) and both the correlated and uncorrelated variables, i.e., \(X\sim\mathcal{N}(0,I)\) and \(X\sim\mathcal{N}(0,\Sigma)\)._  
**Example 4**. : _The non-additive model (from [34])_  
\[f^{*}(\mathbf{x})=(2x_{1}-1)(2x_{2}-1).\]  
_In this case, \(x_{ij}=\frac{w_{ij}+\eta u_{i}}{1+\eta}\), where \(w_{ij}\) and \(u_{i}\) are independently sampled from \(\mathcal{N}(0,1)\). When \(\eta=0\), the input variables \(x_{1}\) and \(x_{2}\) are independent, whereas \(\eta\
eq 0\) means they are correlated. We also consider three cases \((N,p)=(100,10),(100,20)\) and \((200,50)\) with different choices \(\eta=0,0.2,0.5\)._  
As mentioned above, three parameters, including \(E\), \(L\) and \(\beta\), are involved in our algorithm when performing the variable selection. In the experiment, we generally set \(E\) ranging over \([1200,1500]\), \(L\) ranging over \([100,500]\) and \(\beta\) ranging over \([0.015,0.8]\). For the compared algorithms, the corresponding parameters are tuned near the default values. The final values are chosen by cross validation. Finally, each algorithm will give each variable a statistic that measures the importance. For simplicity, we assume that the number of relevant variables \(p_{true}\) is known, and the first \(p_{true}\) variables with largest statistics are selected. Three common metrics are adopted to measure the performance of each algorithm, including TP (the average number of selected truly relevant variables), FP (the average number of selected truly irrelevant variables) and STP (the standard deviation of TP). Generally, a higher TP with a lower FP and STP indicates a better variable-selection algorithm. We repeated the experiments 20 times with the observation set generated in each circumstance. The average variable selection results of each algorithm are presented in Tables 2-4. The optimal results in each circumstance are marked in bold, where DeepVIB (G) means that \(p(z)\) obeys the Gaussian distribution, while DeepVIB (E) indicates the exponential distribution. The above-reported results tell us the following.