# Multi-Modal Low-Data-Based Learning for Video Classification
## 4 SS-CADA-VAE Model
### Cross- and Distribution Alignment  
In a classical variational autoencoder network, there are three main components: encoder, decoder, and reparameterization module. However, in the case of multi-modal low-data-based problems, the main objective is to learn representations from multiplemodalities and project them into a jointly shared space. Therefore, to achieve this, the standard variational autoencoder network is extended for each modality, and an additional mechanism is used to ensure that the output of each modality network is close to each other.  
In the context of multi-modal low-data-based problems, a joint representation can be learned by training separate encoder and decoder networks for each modality. Specifically, the networks for the visual and textual modalities are denoted as \(Enc_{visual}\), \(Enc_{textual}\), \(Dec_{visual}\), and \(Dec_{textual}\), respectively. These networks enable the creation of separate latent representations for each modality, which are \(h_{visual}\) and \(h_{textual}\). By training these networks to be as close as possible to each other in the latent space, the goal is to learn a common representation that captures the underlying relationships between the modalities. This approach allows for the effective fusion of information from multiple modalities, which can lead to improved performance. To achieve this, the cross-alignment loss is proposed by [11], where the latent representation of each modality is decoded by the other modality's decoder. This approach allows for the creation of a joint representation that can better capture the underlying relationships between the visual and textual modalities.  
Here, the cross-alignment loss is defined as follows:  
\[\begin{split}\mathcal{L}_{CA_{1}}&=\sum_{i}^{N} \mid x_{visual}^{(i)}-Dec_{visual}(h_{textual})\mid\\ \mathcal{L}_{CA_{2}}&=\sum_{i}^{N}\mid x_{textual }^{(i)}-Dec_{textual}(h_{visual})\mid\\ \mathcal{L}_{CA}&=\mathcal{L}_{CA_{1}}\text{+} \mathcal{L}_{CA_{2}}\enspace,\end{split} \tag{4}\]  
where \(\mathcal{L}_{CA}\) aims to minimize the reconstruction loss over the (N) mini-batches when latent variables of each modality are decoded with the other modalities' decoders.  
On the other hand, even though the prior distributions of each modality network are standard multivariate Gaussian distributions, the distributions of visual and textual networks may differ and, more importantly, since the two modalities' latent variables need to be projected onto the same joint shared space, approximated posterior distributions of the two modalities' networks need to be as close as possible in 2-Wasserstein distance [51]. The Wasserstein distance-based distribution alignment loss can be given as:  
\[\mathcal{L}_{DA}=\sum_{i}^{N}(||\ \mu_{visual}-\mu_{textual}\mid\mid_{2}^{2} \ +\ \ ||\ \Sigma_{visual}^{1/2}-\Sigma_{textual}^{1/2}\ ||_{Frobenius}^{2})^{1/2}\enspace, \tag{5}\]  
where \(\mathcal{L}_{DA}\) aims to minimize the distance between the approximated posterior distribution of the visual network and the approximated posterior distribution of the textual network, over the (N) mini-batches. Equation (5) uses \(\mu\) and \(\Sigma\) parameters, which are estimated from \(Enc_{visual}\) and \(Enc_{textual}\).