# Multi-Label Weighted Contrastive Cross-Modal Hashing
## 3 The Proposed Methodology
### Multi-Label Weighted Contrastive Learning  
The core idea of contrastive learning is maximizing mutual information (MI) between similar instances. MI measures the correlation between two variables and quantifies the amount of information they share. However, in practical applications, the joint distribution and marginal distribution between two variables are often unknown, which is troublesome for MI calculation. Therefore, contrastive learning typically employs some approximate calculation methods. For example, InfoNCE [55; 56] provides a low-variance estimate of MI for high-dimensional data. However, InfoNCE is not a supervised learning strategy since it is essentially unable to involve category information. To effectively adapt this learning strategy to multi-label cross-modal hashing, we start with the design of cross-modal supervised contrastive loss, then extend it into a multi-label learning task; as a result, we develop multi-label weighted contrastive loss. **Cross-Modal Supervised Contrastive Loss. ** Inspired by [57], we attempt to extend InfoNCE to a cross-modal supervised learning scenario. Firstly, we select instance pairs that share some categories as positive pairs, and then construct intra-modality InfoNCE loss for both image and text modalities, shown as follows:  
\[ \begin{split}\mathcal{L}_{intra-\mathbf{v}}&=\frac{1}{ n}\sum_{i=1}^{n}\frac{1}{n_{l_{i}}-1}\sum_{j=1}^{n}\mathcal{I}_{i\
eq j}\cdot \mathcal{I}_{l_{i}\prec l_{j}>0}\cdot\mathcal{L}_{intra-\mathbf{v}^{\prime}}^{ ij}\\ \mathcal{L}_{intra-\mathbf{v}}^{ij}&=-\log\frac{\exp \left(\mathbf{\hat{x}}_{v_{i}}\cdot\mathbf{\hat{x}}_{v_{j}}/\tau\right)}{\sum_{k=1}^{n }\mathcal{I}_{i\
eq k}\cdot\exp\left(\mathbf{\hat{x}}_{v_{l}}\cdot\mathbf{\hat{x}}_{v_ {k}}/\tau\right)},\\ \mathcal{L}_{intra-\mathbf{t}}&=\frac{1}{n}\sum_{i=1}^{n }\frac{1}{n_{l_{i}}-1}\sum_{j=1}^{n}\mathcal{I}_{i\
eq j}\cdot\mathcal{I}_{l_ {i}\prec l_{j}>0}\cdot\mathcal{L}_{intra-\mathbf{t}^{\prime}}^{ij}\\ \mathcal{L}_{intra-\mathbf{t}}^{ij}&=-\log\frac{\exp \left(\mathbf{\hat{x}}_{t_{i}}\cdot\mathbf{\hat{x}}_{t_{j}}/\tau\right)}{\sum_{k=1}^{n }\mathcal{I}_{i\
eq k}\cdot\exp\left(\mathbf{\hat{x}}_{t_{i}}\cdot\mathbf{\hat{x}}_{t_{ k}}/\tau\right)},\end{split}  \tag{3}\]where \(n_{l_{i}}\) is the total number of samples that share some common labels with \(\mathbf{l}_{i}\), \(\mathcal{I}_{c}\) is a function indicating that if the condition \(c\) is true, \(\mathcal{I}_{c}=1\); otherwise, \(\mathcal{I}_{c}=0\). Combining the above two loss functions, the intra-modality InfoNCE loss is defined as:  
\[\mathcal{L}_{intra}=\mathcal{L}_{intra-v}+\mathcal{L}_{intra-t}. \tag{5}\]  
In an analogous manner, the inter-modality infoNCE losses for image and text are defined as:  
\[ \begin{split}\mathcal{L}_{inter-v}&=\frac{1}{n} \sum_{i=1}^{n}\frac{1}{n_{l_{i}}}\sum_{j=1}^{n}\mathcal{I}_{l_{i}\cdot l_{j}>0 }\cdot\mathcal{L}_{inter-v^{\prime}}^{ij}\\ \mathcal{L}_{inter-v}^{ij}&=-\log\frac{\exp\left( \tilde{\mathbf{x}}_{v_{i}}\cdot\tilde{\mathbf{x}}_{t_{j}}/\tau\right)}{\sum_{k=1}^{n} \exp\left(\tilde{\mathbf{x}}_{v_{i}}\cdot\tilde{\mathbf{x}}_{k_{k}}/\tau\right)},\\ \mathcal{L}_{inter-t}&=\frac{1}{n}\sum_{i=1}^{n} \frac{1}{n_{l_{i}}}\sum_{j=1}^{n}\mathcal{I}_{l_{i}\cdot l_{j}>0}\cdot \mathcal{L}_{inter-t}^{ij},\\ \mathcal{L}_{inter-t}^{ij}&=-\log\frac{\exp\left( \tilde{\mathbf{x}}_{t_{i}}\cdot\tilde{\mathbf{x}}_{v_{j}}/\tau\right)}{\sum_{k=1}^{n} \exp\left(\tilde{\mathbf{x}}_{t_{i}}\cdot\tilde{\mathbf{x}}_{v_{k}}/\tau\right)}.\end{split}  \tag{6}\]  
Accordingly, the inter-modality InfoNCE for multi-label cross-modal hashing learning is defined as:  
\[\mathcal{L}_{inter}=\mathcal{L}_{inter-v}+\mathcal{L}_{inter-t}. \tag{8}\]  
Combining Equations (5) and (8), we obtain the cross-modal supervised contrastive loss as follows:  
\[\mathcal{L}_{con}=\gamma\mathcal{L}_{intra}+(1-\gamma)\mathcal{L}_{inter}, \tag{9}\]  
where \(\gamma\in[0,1]\) is a trade-off factor of two InfoNCE losses. **Multi-Label Weighted Contrastive Loss.