# An Information Theoretic Approach to Privacy-Preserving Interpretable and Transferable Learning
## 2 Mathematical Background
### Review of Membership-Mappings-Based Conditionally Deep Autoencoders  
**Definition 3** (Membership-Mapping Autoencoder [15]). : _A membership-mapping autoencoder, \(\mathcal{G}:\mathbb{R}^{p}\rightarrow\mathbb{R}^{p}\), maps an input vector \(y\in\mathbb{R}^{p}\) to \(\mathcal{G}(y)\in\mathbb{R}^{p}\) such that_  
\[\mathcal{G}(y)\overset{\text{def}}{=}[ \begin{smallmatrix}\mathcal{F}_{1}(Py)& \cdots&\mathcal{F}_{p}(Py)\end{smallmatrix} ]^{T}, \tag{23}\]_where \(\mathcal{F}_{j}\left(j\in\{1,2,\cdots,p\}\right)\) is a Student-\(t\) membership-mapping, \(P\in\mathbb{R}^{n\times p}(n\leq p)\) is a matrix such that the product \(Py\) is a lower-dimensional encoding for \(y\)._  
**Definition 4** (Conditionally Deep Membership-Mapping Autoencoder (CDMMA) [15; 22]). : _A conditionally deep membership-mapping autoencoder, \(\mathcal{D}:\mathbb{R}^{p}\rightarrow\mathbb{R}^{p}\), maps a vector \(y\in\mathbb{R}^{p}\) to \(\mathcal{D}(y)\in\mathbb{R}^{p}\) through a nested composition of finite number of membership-mapping autoencoders such that_  
\[y^{l} =(\mathcal{G}_{l}\circ\cdots\circ\mathcal{G}_{2}\circ\mathcal{G} _{1})(y),\ \forall l\in\{1,2,\cdots,L\} \tag{24}\] \[l^{*} =\arg\ \min_{l\in\{1,2,\cdots,L\}}\ \|y-y^{l}\|^{2}\] (25) \[\mathcal{D}(y) =y^{l^{*}}, \tag{26}\]  
_where \(\mathcal{G}_{l}(\cdot)\) is a membership-mapping autoencoder (Definition 3)._  
CDMMA discovers layers of increasingly abstract data representation with the lowest-level data features being modeled by the first layer and the highest-level data features being modeled by the end layer [15; 22]. An algorithm (stated as Algorithm 2) has been provided in [15; 22] for the variational learning of CDMMA. ```
0: Dataset \(\mathbf{Y}=\left\{y^{i}\in\mathbb{R}^{p}\mid i\in\{1,\cdots,N\}\right\}\); the subspace dimension \(n\in\{1,2,\cdots,p\}\); maximum number of auxiliary points \(M_{max}\in\mathbb{Z}_{+}\) with \(M_{max}\leq N\); the number of layers \(L\in\mathbb{Z}_{+}\). 1:for\(l=1\) to \(L\)do
2: Set subspace dimension associated to \(l\)-th layer as \(n_{l}=\max(n-l+1,1)\). 3: Define \(P^{l}\in\mathbb{R}^{n_{l}\times p}\) such that the \(i\)-th row of \(P^{l}\) is equal to the transpose of eigenvector corresponding to the \(i\)-th largest eigenvalue of a sample covariance matrix of dataset \(\mathbf{Y}\). 4: Define a latent variable \(x^{l,i}\in\mathbb{R}^{n_{l}}\), for \(i\in\{1,\cdots,N\}\), as \[x^{l,i}:=\left\{ \begin{aligned} P^{l}y^{j}&& \text{if }l=1,\\ P^{l}\tilde{y}^{l-1}(x^{l-1,i};\mathbb{M}^{l-1})&& \text{if }l>1\end{aligned} \right.\] (27) where \(\tilde{y}^{l-1}\) is the estimated output of the \((l-1)\)-th layer computed using (22) for the parameters set \(\mathbb{M}^{l-1}=\{a^{l-1},a^{l-1},M^{l-1},\sigma^{l-1},w^{l-1}\}\). 5: Define \(M^{l}_{max}\) as \[M^{l}_{max}:=\left\{ \begin{aligned} M_{max}&& \text{if }l=1,\\ M^{l-1}&&\text{if }l>1\end{aligned} \right.\] (28)
6: Compute parameters set \(\mathbb{M}^{l}=\{a^{l},a^{l},M^{l},\sigma^{l},w^{l}\}\), characterizing the membership mappings associated to the \(l\)-th layer, using Algorithm 1 on dataset \(\left\{\left(x^{l,i},y^{i}\right)\mid i\in\{1,\cdots,N\}\right\}\) with the maximum possible number of auxiliary points \(M^{l}_{max}\). 7:endfor
8:return the parameters set \(\mathcal{M}=\{\{\mathbb{M}^{1},\cdots,\mathbb{M}^{L}\},\{P^{1},\cdots,P^{L}\}\}\). ```  
**Algorithm 2** Variational learning of CDMMA [15; 22]  
**Definition 5** (CDMMA Filtering [15; 22]). : _Given a CDMMA with its parameters being represented by a set \(\mathcal{M}=\{\{\mathbb{M}^{1},\cdots,\mathbb{M}^{L}\},\{P^{1},\cdots,P^{L}\}\}\), the autoencoder can be applied for filtering a given input vector \(y\in\mathbb{R}^{p}\) as follows:_  
\[x^{l}(y;\mathcal{M})=\left\{ \begin{aligned} P^{l}y,&& l=1\\ P^{l}\tilde{y}^{l-1}(x^{l-1};\mathbb{M}^{l-1})&& l\geq 2 \end{aligned} \right. \tag{29}\]Here, \(\hat{y}^{l-1}\) is the output of the \((l-1)\)-th layer estimated using (22).