# An Information Theoretic Approach to Privacy-Preserving Interpretable and Transferable Learning
## 5 Information Theoretic Measures for Privacy Leakage, Interpretability, and Transferability
### A Unified Approach to Privacy-Preserving Interpretable and Transferable Learning
#### 6.2.4 Experimental Design

Taking a domain as the source and another domain as the target, 12 different transfer learning experiments are performed on the four domains associated to the \"Office+Caltech256\" dataset. Each of the 12 experiments is repeated 20 times via creating 20 random train/test splits. In all of the 240 (\(=12\times 20\)) experiments, Algorithm 9 is applied three times with varying values of privacy-loss bound: first with differential privacy parameters as \((d=1,\epsilon=0.01,\delta=1\times 10^{-5})\), second with differential privacy parameters as \((d=1,\epsilon=0.1,\delta=1\times 10^{-5})\), and third with differential privacy parameters as \((d=1,\epsilon=1,\delta=1\times 10^{-5})\). As Algorithm 9 with different values of privacy-loss bound \(\epsilon\) will result in different models, the transfer and multi-task learning models that correspond to the maximum interpretability measure and maximum transferability measure are considered for an evaluation. #### 6.2.5 Reference Methods

This dataset has been studied previously [31; 32; 33; 34; 35; 36] and thus, as a reference, the performances of the following existing methods were considered:

1. ILS (1-NN) [32]: This method learns an Invariant Latent Space (ILS) to reduce the discrepancy between domains and uses Riemannian optimization techniques to match statistical properties between samples projected into the latent space from different domains. 2. CDLS [35]: The Cross-Domain Landmark Selection (CDLS) method derives a domain-invariant feature subspace for heterogeneous domain adaptation. 3. MMDT [34]: The Maximum Margin Domain Transform (MMDT) method adapts max-margin classifiers in a multi-class manner by learning a shared component of the domain shift as captured by the feature transformation. 4. HFA [36]: The Heterogeneous Feature Augmentation (HFA) method learns common latent subspace and a classifier under the max-margin framework. 5. OBTL [33]: The Optimal Bayesian Transfer Learning (OBTL) method employs a Bayesian framework to transfer learning through the modeling of a joint prior probability density function for feature-label distributions of the source and target domains. #### 6.2.6 Results

Tables 4-15 report the results, and the first two best performances have been marked. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Method** & **Feature Type** & **Accuracy (\%)** \\ \hline privacy-preserving maximum interpretability-measure model & VGG-FC6 & 82.6 \\ privacy-preserving maximum transferability-measure model & VGG-FC6 & 82.6 \\ non-private ILS (1-NN) & VGG-FC6 & **83.3** \\ non-private CDLS & VGG-FC6 & 78.1 \\ non-private MMDT & VGG-FC6 & 78.7 \\ non-private HFA & VGG-FC6 & 75.5 \\ non-private OBTL & SURF & 41.5 \\ non-private ILS (1-NN) & SURF & 43.6 \\ non-private CDLS & SURF & 35.3 \\ non-private MMDT & SURF & 36.4 \\ non-private HFA & SURF & 31.0 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Accuracy (in %, averaged over 20 experiments) obtained in _amazon_\(\rightarrow\)_caltech256_ semi-supervised transfer learning experiments. The first and second best performances have been marked. Finally, Table 16 summarizes the overall performance of the top four methods. As observed in Table 16, the maximum transferability-measure model remains as best performing in the maximum number of experiments. The most remarkable result observed is that the proposed methodology, despite being privacy-preserving, ensuring the differential privacy-loss bound to be less than equal to 1 and not requiring access to source data samples, performs better than even the non-private methods.