# A Survey of Full-Cycle Cross-Modal Retrieval: From a Representation Learning Perspective
## 2 Overview of Cross-Modal Retrieval  
Deep learning models have advanced the domain of cross-modal retrieval by addressing the heterogeneity challenge across diverse modalities. These approaches predominantly emphasize techniques including joint subspace learning, feature extraction, interaction, alignment, matching, and fusion. A considerable number of researchers establish fine-grained interactions and multi-level or multi-stage semantic alignments to mitigate the disparities between visual and linguistic modalities. Furthermore, numerous image text pair corpora can be fine-tuned to perform retrieval tasks. Feature extraction serves as the core module of cross-modal retrieval, encoding a raw corpus into embeddings, such as vision embedding and language embedding. By applying deep learning models, a sequence of features can be extracted. In contrast to traditional CNN networks [11] that focus on grid features at the pixel level, more recent approaches have emerged that explore region features in images, such as the Faster-RCNN algorithm proposed by [12]. The widespread paradigms of pre-training and fine-tuning have been motivated by the transformer [13] and BERT [14] architectures. For example, ViT [15] can directly process patch features, while BERT, UniLM [16], RoBERTa [17], T5 [18], BART [19], transformer, and ViT support text encoders. For image encoders, there are a variety of options, including Faster-RCNN, ResNet [20], Visual Dictionary [21], Swin transformer [22], EfficientNet [23], and Linear Projection. Researchers have incorporated pre-training models into cross-modal retrieval systems, modeling the interactions between cross-modal representations. It has been shown that visual concepts in images are critical and complex for cross-modal representations, unlike relationships between words. By extending the BERT model to images and texts, ViLBERT [24] targets region-based object detection and encodes separate sequences of regions using Faster-RCNN. LXMERT [25], similarly to ViLBERT, encodes regions as a sequence of region-of-interest (ROI) features. Apart from region features, pixel-level grid features, such as SOHO [26], CLIP-ViL [27], and pixel-BERT [28], are encoded. They abandon the time-consuming Faster-RCNN. On the contrary, studies are in favor of ResNet extracting grid features. Apart from the region and grid features, patch projection is also used to present image features in many scenarios. ALBEF [29] processes patch features utilizing the ViT encoder directly, generating several flattened 2D patches. OSCAR [30] and ERNIE-ViL [31] develop additional information to facilitate semantic alignments. OSCAR adds region tags as anchor points from images and then implicitly aligns with text words. On the contrary, ERNIE-ViL simulates a scene graph and pays attention to objects with attributions and relations. The two image- and sentence-retrieval scenarios have been extensively studied to align images and texts with the same semantics [32; 33; 34; 35; 36; 37]. At the beginning of a cross-modal alignment study, [PERSON] [32] developed a model that used CNN and Bi-RNN to construct descriptions of pictures and regions. The alignment models incorporate CNN over image regions and bidirectional RNN over sentences. A structured objective leverages a multi-modal embedding to align the two modalities. [PERSON] et al. [33] simultaneously leveraged retrieval and class-guided features and formulated a joint objective function and the loss of classification in a shared latent space. The double loss is considered precisely the retrieval loss and class loss. The double triplet scheme brings forward the novel idea of a loss function for cross-modal research. Some researchers presented a dynamic router schema for interactions between different modalities [34]. They designed a framework with four cells to dynamically align fine-grained segments. ViLT [35] utilized linear projections for matching and demonstrated improvements based on alignment pre-training models,which embed images and captions in the end.