# Variable Selection Using Deep Variational Information Bottleneck with Drop-Out-One Loss
## 1 Introduction  
In many problems of interest, researchers always aim to understand which part of the input variables (features) explain the response variable (output), such as in biomedicine and financial engineering. The variable selection, in addition to reducing the computational complexity, also provides insight into the complex relationship between the inputs and response. Hence, its value is only increasing. To address this problem, many traditional machine-learning models have been studied for decades. For linear models, numerous variable-selection algorithms have been proposed, among which, the most well-known ones include Lasso [1], GroupLasso [2] and Smoothly Clipped Absolute Deviation Penalty (SCAD) [3]. To be specific, [PERSON] [1] realized the variable selection by embedding the \(L_{1}\) regularizer into the linear model. GroupLasso [2] considered the problem of selecting grouped variables for accurate prediction in regression. SCAD [3] extended the thresholding penalty function and satisfied the mathematical conditions for unbiasedness, sparsity and continuity. In these models, the variable importance is measured by the coefficient of each variable. As an extension of the linear model, the additive models have been proposed to deal with nonlinear variable selection problems, such as Sparse Additive Models (SpAM) [4, 5] and GroupSpAM [6]. They, respectively, extended the Lasso and GroupLasso with additive nonparametric regression. Instead of selecting the variable directly, [PERSON] et al. [7] proposed to learn the gradient at each point in the instance space. On this basis, several variants of the gradient-learning (GL) model have been devoted to developing alternativesfor individual purposes, such as sparse [8] and robust [9; 10] gradients. [PERSON] and [PERSON] [8] proposed to learn sparse gradients for variable selection and dimension reduction. To resist the outliers or heavy-tailed noise, [PERSON] et al. [9] proposed robust loss, and [PERSON] et al. [10] adopted tilted loss to learn the gradient function. In these models, the variable importance is measured by the corresponding function. To control the false discovery rate (FDR) of selected variables, [PERSON] et al. [11] introduced a new variable-selection procedure called knockoff filter and achieved exact FDR control in the statistical linear model. On this basis, [PERSON] et al. proposed model-X knockoffs [12] and provided valid inference in a general setting that the conditional distribution of the response is arbitrary and completely unknown. In these models, the variable importance is measured by the difference of scores between the original variable and its knockoff variable. In addition, there also exist several methods proposed from the perspective of information gain, such as decision tree and random forest (RF) [13]. Despite the aforementioned studies having achieved satisfactory performance on variable selection, they are usually limited by the fitting accuracy due to the arrival of the era of big data. In contrast, deep neural networks (DNNs) have achieved great success on a wide variety of tasks due to their strong representation and approximation ability. However, they are largely treated as black-box tools with little interpretability. Therefore, various attempts have been made to uncover the mysteries of neural networks, among which a large class of methods is focused on designing new DNNs to uncover the key features. For example, [PERSON] et al. [14] extended the Lasso and proposed a corresponding DNN framework, LassoNet. It achieved feature sparsity by allowing a feature to participate in a hidden unit only if its linear representative is active. [PERSON] et al. [15] fit the DNN using the sparse group lasso penalty on the first-layer input weights, which resulted in a DNN that only used a subset of the original features. [PERSON] et al. [16] likewise proposed a neural additive model (NAM), which effectively combined the expressivity of DNN with the inherent intelligibility of generalized additive models.