# An Information Theoretic Approach to Privacy-Preserving Interpretable and Transferable Learning
## 5 Information Theoretic Measures for Privacy Leakage, Interpretability, and Transferability
### A Unified Approach to Privacy-Preserving Interpretable and Transferable Learning
12: return in the source domain: classifier \(\{\mathcal{P}_{c}^{\mathrm{g}+\pi}\}_{c=1}^{\mathrm{C}}\); privacy leakage \(IL_{f_{\sum_{s=y_{sr}}^{+}}}\) and adversary model \(\mathbb{BM}^{\mathbb{M}^{\pi r\to x_{sr}}}\); interpretability measure \(IL_{f_{\sum_{s=y_{sr}}^{+}}}\) and interpretability model \(\mathbb{BM}^{\mathbb{M}^{\pi r\to t_{\text{g}}}}\). 13: return in the target domain: classifier \(\{\mathcal{P}_{c}^{\mathrm{g}}\}_{c=1}^{\mathrm{C}}\). 14: return for transfer and multi-task learning scenario: classifiers \(\{\mathcal{P}_{c}^{\mathrm{g}+\pi r}\}_{c=1}^{\mathrm{C}}\) and \(\{\mathcal{P}_{c}^{\mathrm{g}}\}_{c=1}^{\mathrm{C}}\); source2 target model \(\mathbb{M}^{\pi r\to t_{\text{g}}}\); latent subspace transformation matrices \(V^{+\pi r}\) and \(V^{\mathrm{fg}}\); transferability measure

## 6 Experiments

Experiments have been carried out to demonstrate the application of the proposed measures (for privacy leakage, interpretability, and transferability) to privacy-preserving interpretable and transferable learning. The methodology was implemented using MATLAB R2017b and the experiments have been made on an iMac (M1, 2021) machine with 8 GB RAM. ### MNIST Dataset

The MNIST dataset contains \(28\times 28\) sized images divided into a training set of 60,000 images and test set of 10,000 images. The images' pixel values were divided by 255 to normalize the values in the range from 0 to 1. The \(28\times 28\) normalized pixel values of each image were flattened to an equivalent 784-dimensional data vector. #### 6.1.1 Interpretable Parameters

For an MNIST digits dataset, there exist no additional interpretable parameters other than the pixel values. Thus, we defined corresponding to a pixel values vector \(y\in[0,1]^{784}\) an interpretable parameter vector \(t\in\{0,1\}^{10}\) such that the \(j\)-th element \(t_{j}=1\), if the \(j\)-th class label is associated to \(y\); otherwise, \(t_{j}=0\). That is, interpretable vector \(t\), in our experimental setting, represents the class label assigned to data vector \(y\). #### 6.1.2 Private Data

Here, we assume that pixel values are private, i.e., \(x_{sr}=y_{sr}\). #### 6.1.3 Semi-Supervised Transfer Learning Scenario

A transfer learning scenario was considered in the same setting as in [(22; 30)] where 60,000 training samples constituted the source dataset; a set of 9000 test samples constituted the target dataset, and the classification performance was evaluated on the remaining 1000 test samples. Out of 9000 target samples, only 10 samples per class were labeled and the remaining 8900 target samples remained as unlabeled. #### 6.1.4 Experimental Design

Algorithm 9 is applied with the differential privacy parameters as \(d=1\) and \(\delta=1\times 10^{-5}\). The experiment involves six different privacy-preserving semi-supervised transfer learning scenarios with privacy-loss bound values as \(\epsilon=0.1\), \(\epsilon=0.25\), \(\epsilon=0.5\), \(\epsilon=1\), \(\epsilon=2\), and \(\epsilon=10\). For the computation of privacy leakage, interpretability measure, and transferability measure in Algorithm 9, a subset consisting of 5000 randomly selected samples was considered. #### 6.1.5 Results

The experimental results have been plotted in Figure 4. Figure 4a-c display the privacy-accuracy trade-off curve, privacy-interpretability trade-off curve, and privacy-transferability trade-off curve respectively. The following observations are made:

* As expected and observed in Figure 4f, the transferability measure is positively correlated with the accuracy of source-domain classifier on target test samples. * Since we have defined the interpretable vector associated to a feature vector as representing the class label, the positive correlations of interpretability measure with the source domain classifier's accuracy and the transferability measure are observed in Figure 4e and Figure 4f, respectively.