# Multi-Modal Low-Data-Based Learning for Video Classification
## 3 A New Multi-Modal Video Classification Dataset
### Why Do We Need the HVU-LD Dataset?  
The main objective of few-shot learning and zero-shot learning video classification paradigms is to facilitate the training of a learning model using the meta-training set while iteratively refining learning model parameters and hyper-parameters, such as learning rates, weight decay coefficients, and batch sizes, through validation on the meta-validation set. Upon completion of the meta-training and meta-validation phases, the model's performance in generalizing to unseen classes is assessed on the meta-testing set, which comprises novel classes. At this juncture, according to the meta-learning philosophy, the dataset must  
\begin{table}
\begin{tabular}{l l l} \hline \hline
**Category** & **\# of Labels** & **\# of Videos** \\ \hline Scene & 419 & 366,941 \\ Object & 2651 & 480,821 \\ Action & 877 & 481,418 \\ Event & 149 & 320,428 \\ Attribute & 160 & 369,668 \\ Concept & 122 & 375,664 \\ \hline \hline \end{tabular}
\end{table}
Table 1: The overall distribution per category number of labels and the number of videos.  
be split into three groups for meta-training, meta-validation, and meta-testing stages, while taking into account that each group's intersection must be disjoint.  
From this main objective, in the zero-shot video classification task, the model is expected to classify the visual modality samples--videos--in the query set using the auxiliary context--video-level semantic labels--in the support set. The task definition of ZSL is expressed as follows. Let \(D_{s}=\{(x,y,a(y))\,|\,x\in X^{s},y\in Y^{s},a(y)\in A^{s}\}\) is the meta-training set consisting of visual features \(x\), class labels \(y\), and auxiliary context \(a(y)\), where \(s\) denotes the seen classes. At the same time, the meta-testing dataset can be defined as follows: \(D_{u}=\{(x,y,a(y))\,|\,x\in X^{u},y\in Y^{u},a(y)\in A^{u}\}\), where \(u\) denotes the unseen classes and the intersection of \(Y_{s}\), and \(Y_{u}\) is a null set. In the inference stage, the learning model aims to obtain good relations between \(A^{u}\to Y^{u}\) for the query set visual modality samples, \(X^{u}\).  
Similar to the zero-shot setting, few-shot learning includes the \(D_{s}\) and \(D_{u}\) meta-sets, but these sets do not have any auxiliary context, and in the meta-testing phase, the learning model is expected to learn the relationship between \(X^{u}\to Y^{u}\) using visual samples in the support and query sets. An example for a meta-learning episode is given in Appendix A.  
At this point, the first zero-shot learning and few-shot learning settings need a large-scale and high-variation dataset that can mimic real-world scenarios. However, existing datasets [7, 8, 35] in low-data-based video classification are highly biased towards human-oriented actions or specific sports actions and do not cover many other actions in real-world cases. Additionally, as presented above, the zero-shot learning setting requires rich auxiliary context, \(A^{s}\) and \(A^{u}\), since in the inference stage, it is expected to have the \(A^{u}\to Y^{u}\) relation. However, existing datasets only have class labels that refer to actions in the video and unfortunately do not have the additional context to describe the video from different perspectives. For example, the CUB [39] dataset, in the domain of still images, offers 312 binary attributes and 15 part locations, as well as class labels, while the previously existing datasets for the video domain only have class labels. Due to the necessity of such a dataset for the low-data-based video classification problem, we present the HVU-LD dataset for the gap in this research domain.