# A Survey of Full-Cycle Cross-Modal Retrieval: From a Representation Learning Perspective
## 3 Fine-Grained Deep Learning Methodologies
### Cross-Modal Interaction  
Compared to the feature representation, the image text matching strategy improves consistency by investigating semantic relationships. Cross-modal interaction plays a critical role in establishing connections between distinct modal representations. This interaction involves matching each pixel, region, or patch to a specific label. There are three primary approaches in cross-modal interaction, namely vision language alignment, vision language reconstruction, and vision language embedding based on semantic associations. Vision language alignment. Vision language alignment aims to maximize the comparability of image text pairs using large-scale contrastive learning in the dual-encoder model. It employs a re-sharing strategy to address cross-modal heterogeneity between two network branches. In addition, the intra-modal similarity is learned using samples from the exact modal via two conjoined CNN models. In traditional studies, the mode of engagement for cross-modal retrieval has largely relied on manual expert knowledge and empirical input. However, the study [67] proposed a dynamic interaction mechanism for modeling cross-modal retrieval, known as DIME. DIME employs alternative interaction approaches that are tailored to the complexity of the samples. The model includes a local modification cell, an intra-modal inference cell, a global local guidance cell, and a modification cell. ViLT [35] is a novel approach that incorporates visually embedded features via patch projection and patch-level matching of the image and text information. By avoiding the need for time-consuming object identification and convolution techniques with limited expressive capacity, it can effectively improve the performance of cross-modal retrieval. Similarly, in the work by [36], ROSITA employs a pre-training task to enhance fine-grained semantic alignment by suppressing the interference of intra-modal context and eliminating potential noise interference. These advancements demonstrate the effectiveness of these techniques in overcoming the limitations of traditional cross-modal retrieval methods. The ROSITA model draws inspiration from OSCAR and ERNIE-ViL. Additionally, a recent study proposed a new alignment model [68], which embeds images and captions into  
Figure 3: Classified diagram of V-L feature extraction. the same subspace and enhances image-caption retrieval. The ALBEF model [29] adopts the pre-fusion alignment method and utilizes a transformer-based ViT to collect image features without the use of a CNN. The ViT model employs BERT for text and processes single-modal text using the first six layers and multi-modal processing using the last six layers. The model initially performs self-attention on text, followed by cross-attention and visual feature fusion. Moreover, several studies have extensively explored instancewise alignment. For example, X-DETR [37] introduces a versatile architecture for instance-level alignment and discovers that costly joint modal transformers may be redundant for vision language tasks, while weakly annotated data may be beneficial. X-DETR aligns graphics and texts using the dot product. UVLP [69] demonstrates that the combination of image text alignment and the alignment of the entire image text can achieve excellent unsupervised vision language pre-training without parallel data, based on two critical criteria. The authors proposed constructing a weakly supervised paired corpus and granularity alignment pre-training tasks. Their unsupervised pre-training strategy aims to establish robust joint representations of unaligned texts and images, and the results showed admirable performance across various tasks in an unsupervised setting. The aforementioned alignment approaches have specific criteria pertaining to the dataset size and quality and model granularity, which are essential for achieving optimal results. These techniques emphasize the importance of fine-grained matching in cross-modal retrieval. Vision language reconstruction.