# Semi-Supervised Manifold Alignment Using Parallel Deep Autoencoders
## 3 Parallel Deep Autoencoder  
Let \(A:\mathbb{R}^{m}\longrightarrow\mathbb{R}^{p}\) and \(B:\mathbb{R}^{p}\longrightarrow\mathbb{R}^{m}\) be mappings representing the encoder and decoder parts, respectively, of an autoencoder, that maps an input vector \(X\in\mathbb{R}^{m}\) to an output \(\hat{X}=B\circ A(X)\) of equal dimension where \(A(X)\) at the code layer is \(p\)-dimensional with \(p<m\). In a deep autoencoder, both mappings \(A\) and \(B\) comprise several layers [42].  
The autoencoders of the present study use \(p=3\), so that the code layer activations provide a compressed representation of the input that can be visualised as a 3D point cloud, which can approximate a manifold, for example a 2D surface in \(\mathbb{R}^{3}\).  
In our neural network-based approach to manifold alignment, two deep autoencoders are run in parallel (Figure 1). The two encoders \(A_{X}\) and \(A_{Y}\) embed the two datasets \(X\) and \(Y\) in 3D coded space as \(S_{X}=A_{X}(X)\) and \(S_{Y}=A_{Y}(Y)\), respectively. Then, the decoders \(B_{X}\) and \(B_{Y}\) try to reconstruct the data to \(\hat{X}=B_{X}(A_{X}(X))\) and \(\hat{Y}=B_{Y}(A_{Y}(Y))\) in \(\mathbb{R}^{m}\).  
If dataset \(X\) has \(n\) instances or points \(X_{1},\ldots,X_{n}\) and each instance comprises \(m\) features, then \(X\) is represented as a \((n\times m)\)-matrix where the instances correspond to the rows. Training is conducted to minimise the reconstruction errors \(E(X,\hat{X})=(\frac{1}{n}\sum_{i=1}^{n}\|X_{i}-\hat{X}_{i}\|^{2})^{\frac{1}{2}}\) and \(E(Y,\hat{Y})=(\frac{1}{n}\sum_{i=1}^{n}\|Y_{i}-\hat{Y}_{i}\|^{2})^{\frac{1}{2}}\). These two error functions are combined to obtain the total reconstruction error:  
\[E_{recon}=E(X,\hat{X})+E(Y,\hat{Y}). \tag{1}\]  
A correspondence error between the two code layers is calculated to regulate the alignment of the compressed data. \(X_{c}\subset X\) and \(Y_{c}\subset Y\) are defined as the subsets of corresponding points, that is selected pairs of points from both datasets that are in a one-to-one relationship and refer to the same state of the system. In the simulation experiments, \(X_{c}\subset X\) and \(Y_{c}\subset Y\) comprised a uniform selection of \(v\%\) of the data of \(X\) and \(Y\), respectively. When reducing the dimensionality of \(X\) and \(Y\), also \(X_{c}\) and \(Y_{c}\) are mapped into 3D space, resulting in \(D_{X}=A_{X}(X_{c})\subset S_{X}\) and \(D_{Y}=A_{Y}(Y_{c})\subset S_{Y}\). The error between corresponding points in \(D_{X}\) and \(D_{Y}\) is calculated as:  
\[E_{corr}=\sqrt{\frac{100}{v\cdot n}\sum_{x_{i}\in D_{X},y_{i}\in D_{Y}}\|x_{i }-y_{i}\|^{2}} \tag{2}\]  
and we refer to it as the correspondence error. The aim of minimising \(E_{corr}\) is to align the manifolds underlying the data closely in coded space so that they can support each other in establishing the joint latent manifold that underlies both datasets. Finally, the total model error is defined as:  
\[E=\lambda E_{recon}+\mu E_{corr} \tag{3}\]  
Parameters \(\lambda\) and \(\mu\) can be tuned to put more emphasis on the dimensionality reduction aspects or the alignment aspect of the model. The parameter setting \(\lambda=\mu=1\) was sufficient to demonstrate the desired abilities of PDAE for our study. These weight parameters may require careful re-adjustment on other more complex data. By minimising \(E_{recon}\), the deep autoencoders try to learn the intrinsic manifolds in \(X\) and \(Y\) separately. The simultaneous minimisation of \(E_{recon}\) and \(E_{corr}\) reduces the distance of the code layer outputs where a joint three-dimensional representation of the datasets is generated, that is the parallel model trained by minimising the total error \(E\) extracts the intrinsic manifolds at the code layer and simultaneously aligns them.