# An Information Theoretic Approach to Privacy-Preserving Interpretable and Transferable Learning
## 4 Evaluation of the Information-Leakage
### Variational Approximation of the Information Leakage  
The mutual information between \(t\) and \(x\) is given as  
\[I(t;x) =H(t)-H(t|x) \tag{85}\] \[=H(t)+\int p(t,x)\log(p(t|x))\,\mathrm{d}t\,\mathrm{d}x\] (86) \[=H(t)+\left\langle\log(p(t|x))\right\rangle_{p(t,x)} \tag{87}\]  
where \(\left\langle g(x)\right\rangle_{p(x)}\) denotes the expectation of a function of random variable \(g(x)\) with respect to the probability density function \(p(x)\); \(H(t)\) and \(H(t|x)\) are marginal and conditional entropies, respectively. Consider the conditional probability of \(t\) which is given as  
\[p(t|x)=\int\mathrm{d}\theta\,\mathrm{d}\gamma\;p(\theta,\gamma,t|x) \tag{88}\]  
where \(\theta\) is a set defined as in (64). Let \(q(\theta,\gamma)\) be an arbitrary distribution. The log conditional probability of \(t\) can be expressed as\[\log(p(t|x)) =\int\mathrm{d}\theta\,\mathrm{d}\gamma\,q(\theta,\gamma)\log(p(t|x)) \tag{89}\] \[=\int\mathrm{d}\theta\,\mathrm{d}\gamma\,q(\theta,\gamma)\log\!\left( \frac{p(\theta,\gamma,t|x)}{p(\theta,\gamma|t,x)}\right)\] (90) \[=\int\mathrm{d}\theta\,\mathrm{d}\gamma\,q(\theta,\gamma)\log\! \left(\frac{p(\theta,\gamma,t|x)}{q(\theta,\gamma)}\right)+\int\mathrm{d} \theta\,\mathrm{d}\gamma\,q(\theta,\gamma)\log\!\left(\frac{q(\theta,\gamma)}{ p(\theta,\gamma|t,x)}\right)\!. \tag{91}\]  
Define  
\[\mathcal{L}(q(\theta,\gamma),t,x):=\int\mathrm{d}\theta\,\mathrm{d}\gamma\,q( \theta,\gamma)\log\!\left(\frac{p(\theta,\gamma,t|x)}{q(\theta,\gamma)}\right) \tag{92}\]  
to express (91) as  
\[\log(p(t|x))=\mathcal{L}(q(\theta,\gamma),t,x)+\mathrm{KL}(q(\theta,\gamma)\| p(\theta,\gamma|t,x)) \tag{93}\]  
where KL is Kullback-Leibler divergence of \(p(\theta,\gamma|t,x)\) from \(q(\theta,\gamma)\). Using (87),  
\[I(t;x)=H(t)+\left\langle\mathcal{L}(q(\theta,\gamma),t,x)\right\rangle_{p(t,x) }+\left\langle\mathrm{KL}(q(\theta,\gamma)\|p(\theta,\gamma|t,x))\right\rangle _{p(t,x)}. \tag{94}\]  
That is,  
\[=H(t)+\int p(t,x)\log(p(t|x))\,\mathrm{d}t\,\mathrm{d}x\]0  
Since Kullback-Leibler divergence is always non-zero, it follows from (95) that \(\left\langle\mathcal{L}\right\rangle_{p(t,x)}\) provides a lower bound on \(IL_{f_{t\to x}}\) i.e.,  
\[=H(t)+\int p(t,x)\log(p(t|x))\,\mathrm{d}t\,\mathrm{d}x\]1  
Our approach to approximate \(IL_{f_{t\to x}}\) is to maximize its lower bound with respect to variational distribution \(q(\theta,\gamma)\). That is, we seek to solve  
\[=H(t)+\int p(t,x)\log(p(t|x))\,\mathrm{d}t\,\mathrm{d}x\]2  
**Result 1** (Analytical Expression for the Information Leakage). : _Given the model (78)-(81), \(\widehat{IL}_{f_{t\to x}}\) is given as_  
\[=H(t)+\int p(t,x)\log(p(t|x))\,\mathrm{d}t\,\mathrm{d}x\]3 \[=H(t)+\int p(t,x)\log(p(t|x))\,\mathrm{d}t\,\mathrm{d}x\]4 \[=H(t)+\int p(t,x)\log(p(t|x))\,\mathrm{d}t\,\mathrm{d}x\]5 \[=H(t)+\int p(t,x)\log(p(t|x))\,\mathrm{d}t\,\mathrm{d}x\]6  
_Here, \(\Gamma\left(\cdot\right)\) is the digamma function and the parameters \((\bar{\Lambda}_{k},\bar{m}_{k},\bar{a}_{\gamma},\bar{b}_{\gamma})\) satisfy the following:_\[=H(t)+\int p(t,x)\log(p(t|x))\,\mathrm{d}t\,\mathrm{d}x\]7 \[=H(t)+\int p(t,x)\log(p(t|x))\,\mathrm{d}t\,\mathrm{d}x\]8 (100) \[=H(t)+\int p(t,x)\log(p(t|x))\,\mathrm{d}t\,\mathrm{d}x\]9 (101) \[=H(t)+\left\langle\log(p(t|x))\right\rangle_{p(t,x)} \tag{87}\]0  
**Proof of Result 1.