# Variable Selection Using Deep Variational Information Bottleneck with Drop-Out-One Loss
## 1 Introduction
This model learned a linear combination of neural networks that each attended to a single input feature. [PERSON] et al. [17] designed a new DNN architecture (called DeepPINK) with filter-integrated knockoffs, which improved the interpretability and reproducibility of the DNN by incorporating the idea of feature selection with a controlled error rate. On the other hand, recent works also focused on understanding the learning dynamics of DNNs through an information-theoretic lens. An information-theoretic paradigm for deep learning based on the IB framework has caused a great deal of concern as well. [PERSON] and [PERSON] [18] suggested that the goal of the DNN was to optimize the information bottleneck (IB) [19] tradeoff between compression and prediction, successively, for each layer. They claimed that there existed two different and distinct phases: the compression phase and fitting phase in stochastic gradient descent (SGD) optimization. In addition, it was found that most of the training epochs in standard deep learning were spent on compression of the input to efficient representation and not on fitting the training labels. The abstract viewpoint of IB also helps to better understand the field of representation learning, which is an active research area in machine learning that focuses on identifying and uncovering potential explanatory factors [20]. It has attracted increasing attention in many applications, such as out-of-distribution generalization [21], sparse code extraction [22], semi-supervised classification [23] and geometric clustering [24; 25]. However, research from the perspective of IB has rarely been reported for variable selection. In principle, the compression and fitting phases in IB are essential to eliminate useless information. This motivates us to use IB for variable selection. To this end, we investigate the deep variational information bottleneck (DeepVIB) [26] combined with Drop-Out-One loss [27] to realize the variable selection. To better highlight the novelty of the proposed method, we summarize the properties of different variable-selection methods in Table 1, where these methods are categorized as coefficient-based (the variable importance is measured by a coefficient, such as Lasso), function-based (the variable importance is measured by a function, such as additive models and GL), IB-based (through an IB lens, such as the proposed method) and DNN-based (through DNNs, such as Lasso Net and the proposed method). The main contributions of this paper are summarized as below. * **Originality. ** To our knowledge, we are the first to investigate DNNs for variable selection through an IB lens. * **New algorithm. ** A new algorithm based on the DeepVIB and Drop-Out-One loss is designed to realize the variable selection, in which we provide the estimation of the Kullback-Leibler divergence by considering the cases of Gaussian distribution and exponential distribution. The source code is publicly accessed on 21 February 2023 at Github ([[https://github.com/ZeonlungPun/vib_drop_out_one_loss/tree/main](https://github.com/ZeonlungPun/vib_drop_out_one_loss/tree/main)]([https://github.com/ZeonlungPun/vib_drop_out_one_loss/tree/main](https://github.com/ZeonlungPun/vib_drop_out_one_loss/tree/main))). * **Empirical performance. ** Empirical evaluations on simulated and real-world data show that our method performs better than classical variable-selection methods. This confirms the feasibility of variable selection from the perspective of IB. The rest of this paper is organized as follows. Section 2 recalls the background of IB and VIB. Section 3 states the rationality of variable selection with IB and presents its computing algorithm with DeepVIB. Section 4 reports the experimental analysis of our approach. Finally, Section 5 presents our conclusions.