# Variable Selection Using Deep Variational Information Bottleneck with Drop-Out-One Loss
## 3 Proposed Method
### Variable Selection with DeepVIB  
To calculate the statistic \(S_{j}\), we approximate the lower bound in (3) using the empirical data \(D=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{N}\), which is independent and identically distributed (i.i.d.) drawn from (4). We first place all input variables into DeepVIB for training. After convergence, the resulting empirical VIB \(w_{0}\) is calculated as follows:  
\[w_{0}=\frac{1}{N}\sum_{i=1}^{N}\biggl{[}\int p(z|\mathbf{x}_{i})\log q(y_{i}| z)-\beta p(z|\mathbf{x}_{i})\log\frac{p(z|\mathbf{x}_{i})}{r(z)}dz\biggr{]}. \tag{7}\]  
Here, the \(p(z|\mathbf{x}_{i})\) and \(q(y_{i}|z)\) in the first term can be easily obtained by the encoder and decoder of DeepVIB, respectively. As for the second term, the KL-divergence \(KL(p(z|\mathbf{x}_{i})|r(z))\) are determined by the distribution of the latent variable \(r(z)\) and its posterior \(p(z|x)\). In this paper, we consider the Gaussian distribution and the exponential distribution. The detailed proofs of estimating the \(KL(p(z|\mathbf{x}_{i})|r(z))\) are provided in the Appendix A.  
Denoting \(D_{j}=\{(\mathbf{x}_{i-j},y_{i})\}_{i=1}^{N}\) as the data after removing the \(j\)-th variable, we similarly retrain the DeepVIB model and calculate the empirical VIB \(w_{j}\) as follows:  
\[w_{j}=\frac{1}{N}\sum_{i=1}^{N}\left[\int p(z|\mathbf{x}_{i,-j})\log q(y|z)- \beta p(z|\mathbf{x}_{i,-j})\log\frac{p(z|\mathbf{x}_{i,-j})}{r(z)}dz\right],\ j=1,2, \cdots,p.\]  
During the training of DNN, there are several reasons, such as inconsistent initialization of the weight parameters and randomly chosen samples used in each batch, that can cause fluctuations over epochs. To eliminate the randomness and make the statistics stable, a common solution is to calculate the \(w_{0}\) and \(w_{j}\) in the later epochs. We denote the corresponding empirical VIB values at the \(k\)-th epoch as \(w_{0}^{k}\) and \(w_{j}^{k}\), the final VIB values are averaged by the last \(L\) (called the \"reserved length\") epochs, i.e.,  
\[\overline{w_{0}}=\frac{1}{L}\sum_{k=E-L+1}^{E}w_{0}^{k},\ \overline{w_{j}}= \frac{1}{L}\sum_{k=E-L+1}^{E}w_{j}^{k},\]  
where \(E\) means the number of epochs. Then, the empirical statistics approximating (6) are calculated by  
\[\hat{S_{j}}=\overline{w_{0}}-\overline{w_{j}},\ j=1,2,\cdots,p.\]  
The detailed training steps of the proposed method are also summarized in Algorithm 1.  
```
Input :Dataset \(D=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{N}\), parameter \(\beta\), the epochs \(E\), reserved length \(L\), a sequence of scores sets \(W_{j}=\varnothing,j=1,2,\cdots,p\) Output :\(p\) empirical statistics for\(k\)in\(\{1,2,\cdots,E\}\)do  Calculate the empirical VIB with dataset \(D\): \(w_{0}^{k}=\frac{1}{N}\sum_{i=1}^{N}\left[\int p(z|\mathbf{x}_{i})\log q(y_{i} |z)-\beta p(z|\mathbf{x}_{i})\log\frac{p(z|\mathbf{x}_{i})}{r(z)}dz\right]\)  Store \(w_{0}^{k}\) in this epoch: \(W_{0}=W_{0}\cup w_{0}^{k}\)  end for  Calculate the benchmark VIB: \(\overline{w_{0}}=\frac{1}{L}\sum_{k=E-L+1}^{E}w_{0}^{k}\) for each\(j\)in\(\{1,2,\cdots,p\}\)do  Construct the dataset \(D_{j}=\{(\mathbf{x}_{i,-j},y_{i})\}_{i=1}^{N}\) by dropping the \(j\)-th variable for\(k\)in\(\{1,2,\cdots,E\}\)do  Calculate the empirical VIB with dataset \(D_{j}\):  Store \(w_{j}^{k}\) in this epoch: \(W_{j}=W_{j}\cup w_{j}^{k}\)  end for  end for for each\(j\)in\(\{1,2,\cdots,p\}\)do  Calculate the average VIB in the \(L\) later epochs: \(\overline{w_{j}}=\frac{1}{L}\sum_{k=E-L+1}^{E}w_{j}^{k}\)  Score of importance for the \(j\)-th feature: \(\hat{S_{j}}=\overline{w_{0}}-\overline{w_{j}}\)  end for
```  
**Algorithm 1**Calculation algorithm of DeepVIB