# Multi-Modal Low-Data-Based Learning for Video Classification
## 4 SS-CADA-VAE Model
### Implementation Details  
CADA-VAE consists of two encoders and two decoders for each modality and each has one hidden layer. Owing to what the CADA-VAE study proposed for the still-image domain, the input for that model required some adaptations for the video-domain adaptation. In this context, since the video domain has more than one image, unlike the still-image domain in a single sample, input images, also known as frames, are extracted from the video, and then using different feature extractors, visual feature vectors are computed. As a last step, all these visual feature vectors are averaged to create a video-level feature vector. Thus, without a substantial change in the CADA-VAE architecture, it is made applicable to video-domain usage. The overall architecture of our model is illustrated in Figure 2.  
In order to analyze the effect of different visual backbones on the low-data-based video classification problem, two different neural network architectures have been used to obtain videos for frames' visual-level features: Densenet201 [56] and visual transformer [57]. These architectures are pre-trained on ImageNet. And they are completely frozen during the low-data learning process. Regardless of the feature extraction methodology, 16 frames were selected for each video in uniform order and then resized to 224 \(\times\) 224. Then, to obtain the video-level visual feature vector, each of the selected frames is fed into the feature extraction method and their output is averaged. Similarly, regardless of the feature extraction method, the visual hidden layer has 960 neurons in the case of Densenet201 and 384 neurons in the case of ViT usage. Hereby, while the network that uses the Densenet201 feature extraction method has nearly 4 M learning model parameters, the network that uses ViT has nearly 3 M learning model parameters. At the same time, with the same intention, two different pre-trained neural network architectures have been used to obtain videos for frames' textual-level features: ELMO [58] and Bert [59]. While in ELMO, the videos' textual-level feature is constructed by averaging each token's ELMO features, we only use the classification head feature vector in the Bert method. Furthermore, without depending on the textual neural network architecture, the textual hidden layer has 512 neurons to analyze videos' textual-level features. And the latent hidden size for both modalities was set to 64. The result of the number of learning parameters of the textual network has nearly 1.5 M learning model parameters due to both ELMO and Bert models being frozen during the low-data learning process. On the other hand, L2 loss is chosen as a reconstruction and a cross-reconstruction objective. While the reconstruction objective's coefficient is set to 1.0 during the training, the coefficient of the cross-reconstruction, \(\alpha\), increases up to 1.5 between the 20 th and 75 th epochs. And the coefficient of KL-divergence in the ELBO objective is set to 0.25, while the coefficient of the distribution alignment objective, \(\beta\), increases up to 3 between the 10 th and 30 th epochs. Also, the semi-supervision part of the SS-CADA-VAE consists of the \(\gamma\) coefficient. Different combinations and thresholds have been examined and detailed experiments are presented for both the few-shot and the zero-shot classification tasks. All components in the SS-CADA-VAE are trained for 100 epochs by the first-order derivative-based Adam optimizer, while the learning rate is 1 \(\times\) 10\({}^{-3}\). And episodic learning is selected throughout the entire model. In this manner, each episode has \(n\) classes, where each class has \(k\) samples in the support set. These parameters depend on the learning scenario, for example, while _3-way 5-shot_ is one of the few-shot learning options, _3-way 0-shot_ is the zero-shot learning option. In each learning scenario and setting, 15 samples are preferred in the query set, since a low number of test samples may produce biasedobservations.