# An Information Theoretic Approach to Privacy-Preserving Interpretable and Transferable Learning
## 2 Mathematical Background
### Notations  
* Let \(n,N,p,M\in\mathbb{N}\). * Let \(\mathcal{B}(\mathbb{R}^{N})\) denote the _Borel \(\sigma-\)algebra_ on \(\mathbb{R}^{N}\), and let \(\lambda^{N}\) denote the _Lebesgue measure_ on \(\mathcal{B}(\mathbb{R}^{N})\). * Let \((\mathcal{X},\mathcal{A},\rho)\) be a probability space with unknown probability measure \(\rho\). * Let us denote by \(\mathcal{S}\) the set of finite samples of data points drawn i.i.d. from \(\rho\), i.e., \[\mathcal{S}:=\big{\{}(x^{i}\sim\rho)_{i=1}^{N}\mid N\in\mathbb{N}\big{\}}.\] (1)
* For a sequence \(\mathrm{x}=(x^{1},\cdots,x^{N})\in\mathcal{S}\), let \(|\mathrm{x}|\) denote the cardinality, i.e., \(|\mathrm{x}|=N\). * If \(\mathrm{x}=(x^{1},\cdots,x^{N})\), \(\mathrm{a}=(a^{1},\cdots,a^{M})\in\mathcal{S}\), then \(\mathrm{x}\wedge\mathrm{a}\) denotes the concatenation of the sequences \(\mathrm{x}\) and \(\mathrm{a}\), i.e., \(\mathrm{x}\wedge\mathrm{a}=(x^{1},\ldots,x^{N},a^{1},\ldots,a^{M})\). * Let us denote by \(\mathbb{F}(\mathcal{X})\) the set of \(\mathcal{A}\)-\(\mathcal{B}(\mathbb{R})\) measurable functions \(f:\mathcal{X}\to\mathbb{R}\), i.e., \[\mathbb{F}(\mathcal{X}):=\{f:\mathcal{X}\to\mathbb{R}\mid f\text{ is }\mathcal{A}\text{-}\mathcal{B}(\mathbb{R})\text{ measurable}\}.\] (2)
* For convenience, the values of a function \(f\in\mathbb{F}(\mathcal{X})\) at points in the collection \(\mathrm{x}=(x^{1},\cdots,x^{N})\) are represented as \(f(\mathrm{x})=(f(x^{1}),\cdots,f(x^{N}))\). * Let \(\xi_{\mathrm{x}}:\mathbb{R}^{|\mathrm{x}|}\to[0,1]\) be a membership function satisfying the following properties: **Nowhere Vanishing:**\(\xi_{\mathrm{x}}(\mathrm{y})>0\) for all \(\mathrm{y}\in\mathbb{R}^{|\mathrm{x}|}\), i.e., \[\mathrm{supp}[\xi_{\mathrm{x}}]=\mathbb{R}^{|\mathrm{x}|}.\] (3)  
**Positive and Bounded Integrals:** The functions \(\xi_{\mathrm{x}}\) are absolutely continuous and Lebesgue integrable over the whole domain such that for all \(\mathrm{x}\in\mathcal{S}\), we have  
\[0<\int_{\mathbb{R}^{|\mathrm{x}|}}\xi_{\mathrm{x}}\;\mathrm{d} \lambda^{|\mathrm{x}|}<\infty. \tag{4}\]  
**Consistency of Induced Probability Measure:** The membership function induced probability measures \(\mathbb{P}_{\xi_{\mathrm{x}}}\), defined on any \(A\in\mathcal{B}(\mathbb{R}^{|\mathrm{x}|})\), as  
\[\mathbb{P}_{\xi_{\mathrm{x}}}(A):=\frac{1}{\int_{\mathbb{R}^{| \mathrm{x}|}}\xi_{\mathrm{x}}\;\mathrm{d}\lambda^{|\mathrm{x}|}}\int_{A}\xi_{ \mathrm{x}}\;\mathrm{d}\lambda^{|\mathrm{x}|} \tag{5}\]  
are consistent in the sense that  
_where \(\mathsf{x}\in\mathcal{S}\), \(\mathsf{y}\in\mathbb{R}^{|\mathsf{x}|}\), \(\
u\in\mathbb{R}_{+}\setminus[0,2]\) is the degrees of freedom, \(\mathsf{m}_{\mathsf{y}}\in\mathbb{R}^{|\mathsf{x}|}\) is the mean vector, and \(K_{\mathsf{xx}}\in\mathbb{R}^{|\mathsf{x}|\times|\mathsf{x}|}\) is the covariance matrix with its \((i,j)-th\) element given as_  
\[(K_{\mathsf{xx}})_{ij}=kr(x^{i},x^{j}) \tag{9}\]  
_where \(kr:\mathbb{R}^{n}\times\mathbb{R}^{n}\to\mathbb{R}\) is a positive definite kernel function defined as_  
\[kr(x^{i},x^{j})=\sigma^{2}\exp\left(-0.5\sum_{k=1}^{n}w_{k}\Big{|}x ^{i}_{k}-x^{j}_{k}\Big{|}^{2}\right) \tag{10}\]  
_where \(x^{i}_{k}\) is the \(k\)-th element of \(x^{i}\), \(\sigma^{2}\) is the variance parameter, and \(w_{k}\geq 0\) (for \(k\in\{1,\cdots,n\}\))._  
Given a dataset \(\{(x^{i},y^{i})\mid x^{i}\in\mathbb{R}^{n},\ y^{i}\in\mathbb{R}^{p},\ i\in\{1, \cdots,N\}\}\), it is assumed that there exist zero-mean Student-t membership mappings \(\mathcal{F}_{1},\cdots,\mathcal{F}_{p}\in\mathbb{F}(\mathbb{R}^{n})\) such that  
\[y^{i}\approx\big{[}\ \mathcal{F}_{1}(x^{i})\ \cdots\ \mathcal{F}_{p}(x^{i}) \big{]}^{T}. \tag{11}\]  
Under modeling scenario (11), [22] presents an algorithm (stated as Algorithm 1) for the variational learning of membership mappings. ```
0: Dataset \(\{(x^{i},y^{i})\mid x^{i}\in\mathbb{R}^{n},\ y^{i}\in\mathbb{R}^{p},\ i\in\{1, \cdots,N\}\}\) and maximum possible number of auxiliary points \(M_{max}\in\mathbb{Z}_{+}\) with \(M_{max}\leq N\).