# A Survey of Full-Cycle Cross-Modal Retrieval: From a Representation Learning Perspective
## 4 Loss Function

Table 3: _Cont._loss serves as the most-prevalent method for assessing matching models, with random samples, positive examples, and negative examples being essential components to consider. In recent applications of cross-modal retrieval, specific loss functions have been adeptly integrated into model training procedures. In the study conducted on VSE++ [61], hard negative samples were employed to enhance the retrieval methodology in a visual semantic embedding framework. The researchers opted for max of hinges (MH) loss rather than sum of hinges (SH) loss. SH loss accounts for the aggregate of all values, whereas MH loss selects the maximum value among all triplet losses. The modification of conventional loss function applied to multi-modal embedding was motivated by the exploration of hard negatives, the incorporation of hard negative samples in structural prediction, and the rank-based loss function. When combined with fine-tuning strategies and the implementation of augmented data, this approach significantly elevates retrieval performance. In the study of [82], an innovative approach employing adaptive loss was applied to the realm of fashion, facilitating cross-modal retrieval by enabling searches using either text or images. By integrating textual and visual elements into the transformer architecture, adaptive loss was employed to modify the weightings of text matching and image matching. Reference [83] generated images in a feature embedding framework. They proposed a dual-structure embedding that consists of a global semantic embedding and a supplementary local embedding to enhance the performance. The ranking loss is a key component in their approach, with the bidirectional autoencoder effectively learning features in public spaces. Furthermore, they demonstrated that ranking loss with a violation penalty outperforms hard triplet loss in the context of VSE++. A number of scholars employ a diverse range of loss functions in their  
Figure 4: Innovative samples of the loss function. research. These functions may include global consistency loss within a textual modality, local and global loss across modalities, and categorical antagonism loss. Some researchers constructed hybrid networks and precisely defined loss functions, as outlined by [84]. Such functions encompass hash loss, similarity loss, quantization loss, and equalization loss. The aforementioned network facilitates the concurrent training of deep representations and quantizers. The study [85] introduced a comprehensive approach incorporating multiple loss functions to enhance the model's efficacy. These functions encompass semantic differentiation loss, contrastive loss, and significant margin loss. Reference [63] introduced a soft margin triplet loss for noise data processing, addressing noise correspondence. A new polynomial loss function was developed in [86] to determine the polynomial weight with positive and negative samples. This function is capable of selecting informative pairs from redundant pairs. Moreover, the loss functions were discussed separately in [87], and the learning strategy was divided into four parts. With respect to mitigating the semantic discrepancy, the authors introduced a discriminative label function, shown in (Equation (1)):  
\[\ell_{label}=\frac{1}{N}\Big{(}\Big{\|}\varphi^{T}F_{I_{1}},F_{I_{2}},\ldots,F_ {I_{N}}\Big{\|}_{F}+\Big{\|}\varphi^{T}F_{T_{1}},F_{T_{2}},\ldots,F_{T_{N}} \Big{\|}_{F}\Big{)} \tag{1}\]  
where \(F_{I_{i}}\) represents the \(i\)-th image in common subspace and \(F_{T_{i}}\) is the \(i\)-th text's representation. \(N\) denotes the number of image-text pairs. The notation \(\varphi\) is for classification as the parameter vector. \(T\) is the text sample in a multimedia database. The Frobenius norm is defined as \(\|\cdot\|_{F}\). Classifier \(\big{(}F_{I_{i}};\varphi\big{)}\) and Classifier \(\big{(}F_{T_{i}};\varphi\big{)}\) are two classifiers to recognize the semantic discrimination in label space.