# A Survey of Full-Cycle Cross-Modal Retrieval: From a Representation Learning Perspective
## 2 Overview of Cross-Modal Retrieval
[PERSON] [36] was motivated by the highlights of OSCAR and ERNIE-ViL and enhanced alignments by integrating cross- and intra-modal knowledge. In addition, another study [37] offered an instance-oriented vision language task architecture that utilized the dot product to align texts and images. The cross-modal retrieval framework predominantly encompasses fine-grained components: representation, translation, alignment, fusion, and co-learning. This section describes the exact design, comprising essential stages. Figure 2 illustrates the comprehensive architecture of a typical system within this domain. In the full-cycle workflow, these modules are transformed into the following methodologies, including pre-processing, encoder representation, cross-modal attention, and decoder mechanisms. These stages facilitate efficient information extraction and retrieval across different modalities. Pre-processing. The input data are pre-processed to reduce noise and to prepare for subsequent processing. This stage converts image/video and textual phrase inputs into visual and textual tokens. In addition, there are differences between various modalities, so the pre-processing will make a distinction. Apart from the standard tokenization, there are several modules. Encoder representation. The second period involves representing each modality independently using feature extraction methods. The encoder stage gathers input from visual and textual tokens and generates intermediate states to encode semantic content. After embedding, the most-common methodologies to build an encoder are to utilize LSTM, convolution, and other techniques to encode the token sequences. For text representation, word embeddings, positional embeddings, and segment embeddings are fed into the BERT encoder. Additionally, a series of features, such as image representation, is aligned with a text representation. In this scenario, patch, grid, and region features are extracted from the visual domain. Vision language pre-training models combine feature extraction and feature fusion with pre-training tasks. These parts address various challenges, such as quantifying the text and image and transmitting them to the model for learning, handling the challenges of representation interactions, and building pre-training tasks to aid models in learning the alignment information. Pre-training on large-scale data can learn semantic correlation across distinct modalities, addressing the issue of difficult access to pricey manual annotations. There are two core pre-training choices with respect to the fusion encoder and dual encoders to aggregate information in the paired data. The single encoder mainly improves the BERT input, whereas double encoders mainly perform co-/cross-BERT. We examined many current publications from 2018 to 2022 and divided them into single-stream and dual-stream models based on how they treat pre-training models. Table 1 shows a road map for pre-training models with fusion encoder and dual encoders. Studies have shown that the single-stream design conducts self-attention on two modalities directly, overlooking intra-modality interaction. As a result, several researchers advocate the adoption of a dual-stream architecture to describe cross-modal interactions. Figure 2: An overview of the cross-modal retrieval process. A single-stream architecture learns by a single transformer encoder and assumes that it is easy to create a correlation and alignment encompassing vision and language. Furthermore, the features are concatenated with position information before being fed into a transformer-based encoder. Unlike single-stream architectures, dual-stream architectures utilize a cross-modal mechanism to model two unidirectional cross-attention sublayers. The sublayers are typically composed of a cross-attention layer. They are in charge of transferring information and harmonizing semantics. In this case, parameters are shared between two sublayers, and separate transformers learn contextualized embedding information.