# Soft Contrastive Cross-Modal Retrieval
## 4 Experiment
### Setting
#### 4.1.1 Datasets  
We carry out all experiments on three popular datasets, Wikipedia [59], NUS-WIDE [60] and Pascal Sentence [61], which are introduced briefly as follows, and the statistic details of them are shown in Table 2. Besides, the image-text samples of datasets are shown in Figure 5.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Dataset** & **Train Set** & **Test Set** & **Labels** & \(\mathbf{D_{i}}\) & \(\mathbf{D_{t}}\) \\ \hline Wikipedia & 1300 & 1566 & 10 & 4096 & 3000 \\ NUS-WIDE & 8000 & 1000 & 20 & 4096 & 1000 \\ Pascal Sentence & 800 & 100 & 20 & 4096 & 300 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Scale and capacity of the three datasets used in our experiments.  
**Wikipedia.** The Wikipedia dataset ([[https://en.wikipedia.org/wiki/](https://en.wikipedia.org/wiki/)]([https://en.wikipedia.org/wiki/](https://en.wikipedia.org/wiki/)), accessed on 5 December 2023) is composed of 2866 paired image-text documents extracted from \"Wikipedia feature articles\", which are considered to be some of the best content on Wikipedia. These documents are organized into ten semantic categories reflecting diverse interests: art and architecture; biology; geography and places; history; literature and theater; media; music; royalty and mobility; sport and recreation; and warfare. Each document includes a textual description that corresponds directly to the content of the associated image, providing a rich context for multimodal learning tasks. However, there are several notable challenges and potential biases inherent in this dataset: textual description quality, selection bias and representation bias.  
**NUS-WIDE.** The NUS-WIDE dataset ([[https://lms.comp.nus.edu.sg/wp-content/uploads/2019/research/nuswide/NUS-WIDE.html](https://lms.comp.nus.edu.sg/wp-content/uploads/2019/research/nuswide/NUS-WIDE.html)]([https://lms.comp.nus.edu.sg/wp-content/uploads/2019/research/nuswide/NUS-WIDE.html](https://lms.comp.nus.edu.sg/wp-content/uploads/2019/research/nuswide/NUS-WIDE.html)), accessed on 5 December 2023) is a comprehensive collection of 269,648 images sourced from the Flickr platform. Each image of this dataset is accompanied by user-generated tags and cumulatively amounts to 5018 unique labels. This dataset, which serves as input for various image processing tasks, is characterized by its diverse visual content, representing a wide array of scenes, objects, and events. It includes extracted low-level features like color histograms, texture descriptors, and edge direction histograms. Additionally, the dataset provides ground-truth labels for 81 concepts out of the thousands of tags, which have been manually verified, facilitating precise evaluation of image annotation and retrieval algorithms. However, the NUS-WIDE dataset does present several challenges and potential biases: label noise, class imbalance, cultural bias and subjectivity in the ground truth.  
**Pascal Sentence.** The Pascal Sentence Dataset ([[http://vision.cs.uiuc.edu/pascal-sentences/](http://vision.cs.uiuc.edu/pascal-sentences/)]([http://vision.cs.uiuc.edu/pascal-sentences/](http://vision.cs.uiuc.edu/pascal-sentences/)), accessed on 5 December 2023), a subset of the PASCAL Visual Object Classes (VOC), is designed to facilitate pattern analysis, statistical modeling, and computational learning. It encompasses twenty categories: aeroplane, bicycle, bird, boat, bottle, bus, car, cat, chair, cow, dining table, dog, horse, motorbike, person, potted plant, sheep, sofa, train, and TV. The training set comprises 10,103 images, featuring 23,374 objects, averaging approximately 500 objects per category. Some categories have imbalanced representation, which might lead to biases in model performance. For instance, 'person' is a category that is typically over-represented, which can result in models with a disproportionate sensitivity to human figures compared to other categories.