# A Survey of Full-Cycle Cross-Modal Retrieval: From a Representation Learning Perspective
## 2 Overview of Cross-Modal Retrieval
As shown in Table 1, single-stream and dual-stream pre-training models have emerged as two prominent types in recent years. A single-stream architecture operates the unordered representation of transformer attention in a unified framework. A number of studies have used the single-stream paradigm for pre-training, such as VisualBERT [24], UNITER [39], OSCAR [30], Unicoder-VL [42], VL-BERT [43], E2E-VLP [45], ImageBERT [46], Pixel-BERT [28], ViLT [35], VinVL [48], and M6 [50]. Some researchers develop segment embedding from different sources to indicate input elements, i.e., VisualBERT and VL-BERT. The dual-stream models include ViLBERT [24], LXMERT [25], UNIMO [40], ViLLA [41], ALBEF [29], LightningDot [44], CLIP [27], ALIGN [47], ERNIE-ViL [31], WenLan1.0 [17], and COTS [49]. In ViLBERT, the co-transformer handles a two-stream interaction. Moreover, the structure has been updated for interactivity, especially considering the text's context while rendering the image. Furthermore, LXMERT is the same as ViLBERT in the pre-training model. UNIMO brings forth new ideas, which take both a single modal and multiple modals into consideration to make a feature fusion. With respect to ViLLA, it employs adversarial training in the pre-training and fine-tuning stages. Adversarial training can help the model generalize better, enabling the performance at the fine-tuning stage. ALBEF presents two categories, yielding strong single-peak and multi-peak representations with enhanced retrieval and reasoning ability. The study of LightningDot proposes to convert costly attention mechanisms into three types of learning objectives. Cross-modal attention. Much work has been devoted to addressing the representation as mentioned earlier via modeling multi-modal interactions. According to multi-modal representations, correlation modeling is used to learn common representations. The cross-modal interaction encourages other interactions between the two diverse modalities to improve vision language tasks. We classified attention as up-bottom attention, bottom-up attention, recurrent attention, cross-attention, co-attention, distillation-attention, meshed-memory attention, and X-linear attention. The degree of cross-modal information fusion varies among attention mechanisms. Up-bottom attention methods [51] have been widely employed to enable comprehension via fine-grained analysis and even multiple levels \begin{table}
\begin{tabular}{c c} \hline \hline
**Dual-Stream Models** & **Single-Stream Models** \\ \hline
2019-ViLBERT [24] & 2019-VisualBERT [38] \\ \hline
2019-LXMERT [25] & 2020-UNITER [39] \\ \hline
2020-UNIMO [40] & 2020-Oscar [30] \\ \hline
2020-ViLLA [41] & 2020-Unicoder-VL [42] \\ \hline
2021-ALBEF [29] & 2020-VL-BERT [43] \\ \hline
2021-LightningDot [44] & 2020-E2E-VLP [45] \\ \hline
2021-CLIP [27] & 2020-ImageBERT [46] \\ \hline
2021-ALIGN [47] & 2020-Fixed-BERT [28] \\ \hline
2021-ERNIE-ViL [31] & 2021-ViLT [35] \\ \hline
2019-WenLan1.0(RoBERTa-base) [17] & 2021-VinVL [48] \\ \hline
2022-COTS [49] & 2021-M6 [50] \\ \hline \hline \end{tabular}
\end{table} 
Table 1: A road map table for pre-training models with fusion encoder and dual encoders. of reasoning. The bottom-up process suggests picture areas, each with its feature vector, while the up-bottom mechanism sets feature weights. According to the study by [52], image text retrieval uses iterated operations and correspondences between visuals and words via repeated alignment stages with recurrent attention memory. This research gains a deeper understanding of fragment correspondences by exploring the attention mechanism. This understanding is compatible with intricate semantics, suggesting using the complex relationship gradually between images and words. Cross-attention conveys encoder and decoder information in [14]. Transformer tracking [53] (TransT) avoids falling into a local optimum of semantic information algorithms. To solve this issue of constructing high-accuracy tracking systems, TransT introduces a unique attention-based feature fusion network.