# Variable Selection Using Deep Variational Information Bottleneck with Drop-Out-One Loss
## 3 Proposed Method
### Variable Selection with IB  
Let \(\mathcal{X}\subset\mathbb{R}^{p}\) be a compact input space and \(\mathcal{Y}\subset\mathbb{R}\) be an output space. Denote \((X,Y)\) as the pair of explanatory and response variables taking values in \(\mathcal{X}\times\mathcal{Y}\). Assume  
\[Y=f^{*}(X)+\epsilon, \tag{4}\]  
where \(f^{*}:\mathcal{X}\rightarrow\mathcal{Y}\) is an intrinsic target function and \(\epsilon\) is a random noise satisfying some certain conditions, e.g., the zero-mean noise assumption or the zero-mode noise condition. Let \(X_{j}\) be the \(j\)-th feature of \(X\). We also assume that \(f^{*}(X)\) depends on \(X\) only through \(\{X_{j}:j\in T\}\), where \(T\) is an index set of relevant variables and \(|T|<p\) is the cardinality of \(T\). Let \(T^{c}=\{1,2,\cdots,p\}-T\) be an index set of irrelevant variables.  
In another perspective [18], the IB objective is looking for a sufficient statistic \(Z_{0}\), which is a map of \(X\), and extracts all the information that \(X\) has on \(Y\), i.e., satisfying  
Figure 1: The network structure of DeepVIB. The encoder \(p(z|x)\) learns a representation \(Z\) from \(X\), in which the reparameterization trick \(z_{i}=\mu_{i}+\sigma_{i}\odot\epsilon\) is implemented, and \(\odot\) means the element-wise product. The decoder \(q(y|z)\) uses \(Z\) to predict \(Y\).  
\(I(Z_{0},Y)=I(X,Y)\). Since the exact sufficient statistics only exist for very special distributions, the IB framework only provides for finding approximate sufficient statistics. That means \(I(Z_{0},Y)\) captures as much as possible of \(I(X,Y)\)--not necessarily all of it [18]. Let \(Z_{0}\) be the sufficient statistic, and this means  
\[Z_{0}=\operatorname*{arg\,max}_{Z}\{I(Z,Y)-\beta I(X,Z)\}.\]  
Given a Markov chain \(Y\to X\to Z_{0}\)[19], the elementary data processing inequality ensures  
\[I(Z_{0},X)\geq I(Z_{0},Y).\]  
Combining the above inequalities, it is trivial to find that  
\[\{I(Z_{0},Y)-\beta I(Z_{0},X)\}\leq(1-\beta)I(X,Y),\;0<\beta<1. \tag{5}\]  
Inequality (5) provides the intuition that the IB objective can be approximated by \(I(X,Y)\). Denote \(X_{-j}\) as the rest of \(p-1\) variables after removing the \(j\)-th variable. Note that the \(I(X_{j},Y)\) and conditional mutual information \(I(X_{j},Y|X_{-j})\) can be assumed as 0 for any \(j\in T^{c}\); the equation \(I(X,Y)=I(X_{-j},Y)+I(X_{j},Y|X_{-j})\) informs us \(I(X,Y)=I(X_{-j},Y)\). Similarly, the assumptions \(I(X_{k},Y)>0\) and \(I(X_{k},Y|X_{-k})>0\) imply that \(I(X,Y)>I(X_{-k},Y)\) for any \(k\in T\). This means that \(I(X,Y)\) will decrease after removing the relevant variable, while it will not change for irrelevant variables. Together with inequality (5), a natural idea is that the IB objective will decrease when dropping the critical variable. Therefore, we propose a new statistic to measure the importance of the \(j\)-th variable  
\[S_{j}=\max_{Z}\{I(Z,Y)-\beta I(X,Z)\}-\max_{Z}\{I(Z,Y)-\beta I(X_{-j},Z)\},j=1,2,\cdots,p. \tag{6}\]  
A small \(S_{j}\) means the \(j\)-th variable \(X_{j}\) tends to be irrelevant.  
**Remark 1**.: _Intuitively, \(I(X,Z)\) measures the useful information of the inputs extracted by the encoder. It is inversely related with the compression, i.e., the greater the compression, the smaller the \(I(X,Z)\). Combined with \(I(Z,Y)\) denoting predictive power, the IB can be seen as extracting the key information for predicting \(Y\) from \(X\). Therefore, utilizing the \"information\" in the IB to identify the irrelevant variables is sensible._