# A Survey of Full-Cycle Cross-Modal Retrieval: From a Representation Learning Perspective
## 6 Benchmark Datasets
### Cross-Modal Datasets  
Benchmark datasets are vital for evaluating the performance of cross-modal retrieval. They enable researchers to compare and contrast different techniques and algorithms under consistent conditions, providing a comprehensive and accurate assessment of their strengths and limitations. A high-quality dataset that includes multi-modal data, high-quality labels, and diverse content is crucial for training and evaluating models and advancing relevant research. Nonetheless, conventional datasets might be tailored for a particular task, thereby constraining their versatility in addressing different downstream tasks. The high-quality datasets should encompass a broad range of contextual information and scenarios to ensure the model can learn from multiple domains and exhibit robustness in cross-modal retrieval. Researchers create multi-modal datasets to evaluate the efficacy of the proposed cross-modal methodologies. There are some publicly available datasets in cross-modal retrieval that have been obtained through large-scale collection and labeled. \begin{table}
\begin{tabular}{c c l} \hline \hline
**Name** & **Number** & **Description** \\ \hline Flickr 30k [(99)] & 31,783 & The images are accompanied by 1,58,915 captions gathered through crowdsourcing. \\ \hline MS-COCO [(100)] & 3,28,000 & The dataset contains 25,00,000 annotated occurrences. The collection contains artifacts from 91 different categories. \\ \hline Visual Genome [(101)] & 108,077 & There are about 5.4 million captions given to picture areas. It contains around 2.8 million labels for object properties in the picture and approximately 2.3 million labels for object connections. \\ \hline PKU FG-XMedia [(102)] & 50,000 & Over 50,000 samples are included in the PKU FG-XMedia collection, comprising 11,788 graphics, 8000 texts, 18,350 videos, and 12,000 audios. It has different media kinds, a precise granularity of categories, and multiple data sources. \\ \hline Conceptual Caption [(103)] & 3,000,000 & The photographs and descriptions are gathered from the Internet, and they feature a diverse spectrum of emotions. The captions are derived from the HTML alt element of each picture. \\ \hline Objects 365 [(104)] & 630,000 & The collection includes 630,000 photos from 365 categories and up to 10 million frames. It is distinguished by its cast scale, excellent quality, and good generalizability. \\ \hline [[https://storage.google.com/openimages/web/index.html](https://storage.google.com/openimages/web/index.html)]([https://storage.google.com/openimages/web/index.html](https://storage.google.com/openimages/web/index.html)) (accessed on 1 January 2022) & 9,000,000 & Open Images is a library of 9 million collection including image-level labels, 15,851,536 bounding boxes, 2,785,498 segmentation masks, visual connections, 675,155 localized narratives, and 59,919,574 image-level labels. \\ \hline M6 [(50)] & 100,000,000 & It is the most-extensive dataset of a Chinese multi-modal pre-training model, with 1.9 T pictures and 292 G texts. \\ \hline Conceptual 12M [(105)] & 12,000,000 & It includes a much greater spectrum of visual concepts than the Conceptual Captions dataset, used for image captioning model pre-training and end-to-end training. \\ \hline \hline \end{tabular}
\end{table} 
Table 4: _Cont._For example, the NUS-WIDE [92] dataset is frequently used in cross-modal hashing. Researchers can investigate the difficulties of image annotation and retrieval research based on NUS-WIDE. ImageNet ([[https://www.image-net.org/](https://www.image-net.org/)]([https://www.image-net.org/](https://www.image-net.org/)), accessed on 1 January 2022) is for items in an image to have multiple appearances, positions, views, postures, background clutter, and occlusion. MS-COCO [100] contains more images in each category than the ImageNet dataset, allowing for more scenarios to obtain. The MS-COCO dataset only annotates 80 object categories and does not describe all objects in the image. In the traditional scenario, there might be more object categories.