# Soft Contrastive Cross-Modal Retrieval
## 1 Introduction  
Cross-modal retrieval [1; 2] is a hot research spot in Natural Language Processing, especially with the explosive increase in multimedia data (text, image, video, and audio) and application procedures (e.g., Tiktok, Twitter, and YouTube). It intends to retrieve one modality based on another modality, which will support users to acquire various types of information in different categories of multimedia data. The top challenge of cross-modal retrieval is how to effectively embed multimodal representation [3; 4; 5] in one share space and measure the similarities of multimodal data, which aims to bridge the heterogeneity gap in general. Based on the existing approaches [6; 7; 8], a common method to diminish the heterogeneity gap between different types of data and bridge the relation of objects is to construct a common embedding subspace [9; 10; 11; 12], where the multimedia data can be transformed into the same dimensional representation. In this way, the similarity of samples can be calculated directly and accurately. The traditional methods utilize statistical analysis approaches, e.g., CCA [13] and MCCA [14] analyze cross-modal correlation or learn semantic representation by finding linear associations between different datasets, which are the theory foundation of most traditional models. However, these methods are restricted by the linear function of statistics because they cannot acquire non-linear semantic mapping. As deep neural networks (DNNs) [15; 16; 17] have grown by leaps and bounds in recent years, methods based on deep learning promote non-linear representation learning [18; 19; 20; 21] under the strong power of DNNs. **Motivation. ** To address the current limitations in DNNs representation learning, particularly in the context of multimodal data processing, we revisit the conventional approach of leveraging DNNs for Natural Language Processing. Furthermore, that greatly weakens the generalizability of models and directly obstructs realistic application. Additionally, there is a prevailing and questionable assumption in recent research that datasets are free of noise, with image-text pairs being perfectly aligned and instance ground truths being completely accurate. Such an assumption is far from realistic in real-world scenarios, where noise and inaccuracies are often inherent in the data. These inaccuracies can lead to overfitting and poor model performance on unseen or noisy data, which is a common occurrence in practical applications. **Method. ** To handle this restriction of the current model, this article proposes a novel end-to-end framework, named Soft Contrastive Cross-Modal Retrieval (SCCMR), which integrates a deep cross-modal model with soft contrastive learning and smooth label learning to boost common subspace embedding and improve the generalizability and robustness of the model. **Contributions. ** The contributions of our work are shown as follows:  
* We propose a novel end-to-end cross-modal retrieval model, termed the Soft Contrastive Cross-Modal Retrieval (SCCMR), which aims to combine soft contrastive learning and cross-modal learning. To the best of our knowledge, this is the first work to fuse soft contrastive learning into cross-modal retrieval to improve multimodal feature embedding. * To solve the sharp and tortuous margin problem of feature embedding, we use soft contrastive learning for common subspace learning, which assists the model in finding the smooth boundary of embedding representation. To balance the noise data, we utilize smooth label cross-modal learning, which promotes the robustness of the model on real multimedia data. * We carry out extensive experiments on three benchmark multimedia datasets, Wikipedia, NUS-WIDE, and Pascal Sentence. Results illuminate that our proposed method can outperform the current state-of-the-art methods in cross-modal retrieval, which demonstrates the excellent effectiveness of the proposed method. **Roadmap. ** The rest of this article is organized as follows.