# An Information Theoretic Approach to Privacy-Preserving Interpretable and Transferable Learning
## 4 Evaluation of the Information-Leakage
### An Algorithm for the Computing of Information Leakage  
Result 1 forms the basis of Algorithm 8 that computes the information leakage using available data samples.  
```
0: Dataset \(\left\{(x^{i}\in\mathbb{R}^{n},t^{i}\in\mathbb{R}^{q})\mid x^{i}=f_{t\to x}(t^{ i}),\;i\in\{1,\cdots,N\}\right\}\).
1: Apply Algorithm 7 on \(\left\{(x^{i},t^{i})\mid i\in\{1,\cdots,N\}\right\}\) with \(M_{max}=\min(\lceil N/2\rceil,1000)\) (i.e., constraining the maximum possible number of auxiliary points \(M_{max}\) below \(1000\) for computational efficiency) to obtain variational membership-mappings Bayesian model \(\mathbb{BM}^{x\to t}=\{\{\hat{m}_{k},\hat{\Lambda}_{k}\mid k\in\{1,\cdots,q\} \},\hat{a}_{\gamma},\hat{b}_{\gamma}\}\).
2: Initialize \(\bar{a}/\bar{b}\), e.g., as \(\bar{a}/\bar{b}=\bar{a}/\bar{b}\).
3:repeat
4: Update \(\{\bar{\Lambda}_{k},\hat{m}_{k}\mid k\in\{1,\cdots,q\}\},\bar{b}\) using (99)-(102) where expectations \(<\cdot>_{p(x)}\) and \(<\cdot>_{p(t,x)}\) are approximated via sample averages.
5:until convergence.
6: Compute \(\widehat{IL}_{f_{t\to x}}\) using (98) where expectations \(<\cdot>_{p(x)}\) and \(<\cdot>_{p(t,x)}\) are approximated via sample averages.
7:return\(\widehat{IL}_{f_{t\to x}}\) and the model \(\mathbb{BM}^{x\to t}\).
```  
**Algorithm 8** Estimation of information leakage, \(IL_{f_{t\to x}}=I(t;x)-H(t)\), using variational approximation  
The functionality of Algorithm 8 is as follows:  
* Step 1 applies Algorithm 7 for the inference of a variational membership-mappings Bayesian model.
* The loop between step 3 and step 5 recursively estimates the parameters \(\left(\{\bar{\Lambda}_{k},\hat{m}_{k}\mid k\in\{1,\cdots,q\}\},\bar{a},\bar{b}\right)\) using update rules (99)-(102).
* Step 6 computes the information leakage using (98).  
**Remark 4** (Computational Complexity).: _The computational complexity of Algorithm 8 is asymptotically dominated by the computation of inverse of \(M\times M\) dimensional matrix \(\bar{\Lambda}_{k}\) in (100) to calculate \(\bar{m}_{k}\). Thus, the computational complexity of Algorithm 8 is given as \(\mathcal{O}(M^{3})\), where \(M\) is the number of auxiliary points._  
**Example 1** (Verification of Information Leakage Estimation Algorithm).: _To demonstrate the effectiveness of Algorithm 8 in estimating information leakage, a scenario is generated where \(t\in\mathbb{R}^{10}\) and \(x\in\mathbb{R}^{10}\) are Gaussian distributed such that \(x=t+\omega\); \(t\sim\mathcal{N}(0,5I_{10})\); \(\omega\sim\mathcal{N}(0,\sigma I_{10})\) with \(\sigma\in[1,15]\). Since the data distributions in this scenario are known, the information leakage can be theoretically calculated and is given as_  
\[IL_{f_{t\to x}}=5\log(1+5/\sigma)-0.5\log(|(2\pi\mathrm{e}5I_{10})|).\]For a given value of \(\sigma\), 1000 samples of \(t\) and \(x\) were simulated and Algorithm 8 was applied for estimating information leakage. The experiments were carried out at different values of \(\sigma\) ranging from \(1\) to \(15\).  
Figure 3 compares the plots of estimated and theoretically calculated values of information leakage against \(\sigma\). A close agreement between the two plots in Figure 3 verifies the effectiveness of Algorithm 8 in estimating information leakage without knowing the data distributions.